#1)# #int (x^2-3x)*(cosx)^2*dx#
=#1/2int (x^2-3x)*(1+cos2x)*dx#
=#1/2int (x^2-3x)*dx#+#1/2int (x^2-3x)*cos2x*dx#
#A=int (x^2-3x)*dx=x^3/3-(3x^2)/2+2C#
#B=int (x^2-3x)*cos2x#
=#(x^2-3x)*1/2sin2x-(2x-3)(-1/4cos2x)+2(-1/8sin2x)#
=#(2x^2-6x-1)/4*sin2x+(2x-3)/4*cos2x#
Thus,
#int (x^2-3x)*(cosx)^2*dx#
=#1/2*A+1/2*B#
=#x^3/6-(3x^2)/4+(2x^2-6x-1)/8*sin2x+(2x-3)/8*cos2x+C#