Here ,
I=intarc cosx*xdx
Using Integration by parts:
I=arc cosx intxdx-int(d/(dx)(arc cosx)intxdx)dx
I=arc cosx(x^2/2)-int(-1)/sqrt(1-x^2)xxx^2/2dx
I=x^2/2arc cosx+1/2intx^2/sqrt(1-x^2)dx
color(red)(I=x^2/2arc cosx+1/2I_1............to(A)
where, I_1=intx^2/sqrt(1-x^2)dx
Subst. color(blue)(x=sinu=>dx=cosudu
So,
I_1=intsin^2u/sqrt(1-sin^2u)cosudu
I_1=intsin^2u/cosucosudu
I_1=intsin^2udu
I_1=int(1-cos2u)/2du
I_1=1/2[u-(sin2u)/2]+c
I_1=1/2[u-sinucosu]+c
I_1=1/2[u-sinusqrt(1-sin^2u)]+c
Subst. back color(blue)(sinu=x and u=arcsinx
I_1=1/2[arc sinx-xsqrt(1-x^2)]+c
Subst. value of I_1 into color(red)((A) we get
I=x^2/2arc cosx+1/2{1/2[arc sinx-xsqrt(1-x^2)]}+C
I=x^2/2arc cosx+1/4arc sinx-x/4sqrt(1-x^2)+C