int f(x)*g'(x) dx = f(x)*g(x) - int f'(x)*g(x) dx
To integrate by parts, you need to identify what is f(x) and g'(x), i.e. which side to integrate and which side to differentiate.
Now, polynomials, differentiated a finite number of times, will eventually become zero. Sinusoidal functions, on the other hand, can be integrated or differentiated repeated with ease.
In this problem, you need to integrate by parts twice, because you need to differentiate x^2 twice before it goes away. First, let f(x)=x^2 and g'(x)=sin(2x), i.e. differentiate x^2 and integrate sin(2x).
int_0^{pi/2} x^2*sin(2x) dx = -1/2 int_0^{pi/2} x^2*frac{d}{dx}(cos(2x)) dx
= -1/2 ([x^2*cos(2x)]_0^{pi/2} - int_0^{pi/2} cos(2x)*frac{d}{dx}(x^2) dx)
= -1/2 [(pi/2)^2*(-1) - (0)^2*(1)]
+ 1/2 int_0^{pi/2} cos(2x)*(2x) dx
= frac{pi^2}{8} + int_0^{pi/2} x*cos(2x) dx
Now for the second integration by parts.
frac{pi^2}{8} + int_0^{pi/2} x*cos(2x) dx = frac{pi^2}{8} + 1/2 int_0^{pi/2} x*frac{d}[dx}(sin(2x)) dx
= frac{pi^2}{8} + 1/2([x*sin(2x)]_0^{pi/2} - int_0^{pi/2} frac{d}[dx}(x)*sin(2x) dx)
= frac{pi^2}{8} + 1/2 [0-0] - 1/2 int_0^{pi/2} sin(2x) dx
= frac{pi^2}{8} - 1/2 int_0^{pi/2} sin(2x) dx
= frac{pi^2}{8} - 1/2 [-frac{cos(2x)}{2}]_0^{pi/2}
= frac{pi^2}{8} + 1/4 [cos(pi) - cos(0)]
= frac{pi^2}{8} - 1/2
