that's very ugly and would need a load of IBP so i would start by making it even bigger!! using cos 2A = 2 cos^2 A - 1
I = int x^3 cos^2x \ dx
= int x^3 (cos 2x +1 )/2 \ dx
implies 2I = int x^3 cos 2x +x^3 \ dx
implies 2I - x^4/4 = int x^3 cos 2x \ dx qquad triangle ......taking the easy bit to the LHS
so now we IBP the remaining bit over and over, reducing the x^3 term each time
int \ x^3 cos 2x \ dx = int \ x^3 d/dx (1/2 sin 2x) \ dx
which by IBP....
= x^3 /2 sin2x - int \ 3x^2 1/2 sin 2x \ dx
= x^3 /2 sin2x - (3)/2 int \ x^2 sin 2x \ dx
= x^3 /2 sin2x - (3)/2 int \ x^2 d/dx( - 1/2 cos 2x) \ dx
= x^3 /2 sin2x + (3)/4 int \ x^2 d/dx( cos 2x) \ dx
= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - int \ 2x cos 2x \ dx)
and on we go!!
= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - int \ 2x d/dx (1/2 sin 2x ) \ dx)
= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - int \ x d/dx ( sin 2x ) \ dx)
= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - (x sin 2x - int \ sin 2x \ dx))
= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - (x sin 2x + 1/2 cos2x )) + C
= x^3 /2 sin2x + (3)/4 x^2 cos 2x - 3/4 x sin 2x - 3/8 cos2x + C
so back to triangle
implies 2I - x^4/4 = x^3 /2 sin2x + (3)/4 x^2 cos 2x - 3/4 x sin 2x - 3/8 cos2x + C
implies I = x^4/8 + x^3 /4 sin2x + (3)/8 x^2 cos 2x - 3/8 x sin 2x - 3/16 cos2x + C
once again , the key is IBP:
int \ u(x) * d/dx(v(x)) \ dx = u(x) * v(x) - int \ d/dx(u(x)) * v(x) \ dx
