How do you use substitution to integrate #x^2sqrt(4x+9)dx#?

Use the substitution #u=4x+9# so that #du=4dx# and #x=(u-9)/4=1/4 u-9/4#. Then

#int x^2sqrt(4x+9)\ dx=1/4 int ((u-9)/4)^2sqrt(u)\ du#

#=1/64 int (u^2-18u+81)u^{1/2}\ du#

#=1/64 int (u^{5/2}-18u^{3/2}+81u^{1/2})\ du#

#=1/64*2/7 u^{7/2}-9/32*2/5 u^{5/2}+81/64*2/3 u^{3/2}+C#

#=(1/224 u^2-9/80 u+27/32) u^{3/2}+C#

Since #u^2=(4x+9)^2=16x^2+72x+81#, we can write this as

#=(1/14 x^2+9/28x+81/224-9/20x-81/80+27/32)(4x+9)^{3/2}+C#

#=\frac{10x^2-18x+27}{140}(4x+9)^{3/2}+C#

You can rewrite this carefully.

#= intx^2sqrt((2sqrtx)^2 + 3^2)dx#

Now it looks like #sqrt(a^2 + x^2)#, which resembles #sqrt(1 + tan^2theta)#.

Let:
#2sqrtx = 3tantheta#
#x = (9tan^2theta)/4#
#x^2 = (81tan^4theta)/16#
#sqrt((2sqrtx)^2 + 3^2) = sqrt(9tan^2theta + 9) = 3sectheta#
#dx = 9/4*2tanthetasec^2thetad theta = 9/2tanthetasec^2thetad theta#

#=> int(81tan^4theta)/16 3sectheta 9/2tanthetasec^2thetad theta#

#= 2187/32inttan^5thetasec^3thetad theta#

Notice how that constant is irreducible... Ugh. It looks a bit crazy, but it's just large numbers. Using some identities:
#= 2187/32int(sec^2theta - 1)^2sec^2thetasecthetatanthetad theta#

And some u-substitution. Let:
#u = sectheta#
#du = secthetatanthetad theta#

#= 2187/32int(u^2 - 1)^2u^2du#

#= 2187/32int(u^4 - 2u^2 + 1)u^2du#

#= 2187/32intu^6 - 2u^4 + u^2du#

#= 2187/32 (u^7/7 - 2/5u^5 + u^3/3)#

#= 2187/32 (sec^7theta/7 - 2/5sec^5theta + sec^3theta/3)#

Draw a right triangle if you want. Since #tantheta = (2sqrtx)/3#, and #sqrt((2sqrtx)^2 + 3^2) = sqrt(4x + 9)#, #sectheta = sqrt(4x+9)/3#:

#= 2187/32 ((4x+9)^(7/2)/(7*3^7) - 2/(5*3^5)(4x+9)^(5/2) + (4x+9)^(3/2)/(3*3^3))#

#= 2187/32 ((4x+9)^(7/2)/(15309) - 2/(1215)(4x+9)^(5/2) + (4x+9)^(3/2)/(81))#

#= 1/32 ((4x+9)^(7/2)/(7) - 18/(5)(4x+9)^(5/2) + 27(4x+9)^(3/2))#

#= 1/32(4x+9)^(3/2)[(4x+9)^(2)/7 - 18/5(4x+9) + 27] + C#

Now let's figure out how to make this look nicer. Get some common denominators, subtract in the numerator, and factor out #8/35#:
#= 1/32(4x+9)^(3/2)[(16x^2 + 72x + 81)/7 - 72/5x - 162/5 + 27]#

#= 1/32(4x+9)^(3/2)[(80x^2 + 360x + 405 - 504x - 1134 + 945)/35]#

#= 1/32(4x+9)^(3/2)[(80x^2 - 144x + 216)/35]#

#= (1*8)/(32*35)(4x+9)^(3/2)[10x^2 - 18x + 27]#

#= 1/140(4x+9)^("3/2")[10x^2 - 18x + 27] + C#