We can integrate by parts using the logarithm as integral part, so that in the resulting integral we have a rational function:
int x^2ln^2xdx = int ln^2x d(x^3/3)
int x^2ln^2xdx = (x^3ln^2x)/3 - 1/3 int x^3d(ln^2x)
int x^2ln^2xdx = (x^3ln^2x)/3 - 2/3 int x^3 lnx/xdx
int x^2ln^2xdx = (x^3ln^2x)/3 - 2/3 int x^2lnxdx
Solve the resulting integral by parts again:
int x^2lnxdx = int lnx d(x^3/3)
int x^2lnxdx = (x^3lnx)/3 - 1/3 int x^3 d(lnx)
int x^2lnxdx = (x^3lnx)/3 - 1/3 int x^3 dx/x
int x^2lnxdx = (x^3lnx)/3 - 1/3 int x^2 dx
int x^2lnxdx = (x^3lnx)/3 - 1/9x^3 +C
Substituting in the first expression:
int x^2ln^2xdx = (x^3ln^2x)/3 - 2/3( (x^3lnx)/3 - 1/9x^3 ) +C
and simplifying:
int x^2ln^2xdx = (x^3ln^2x)/3 - 2/9(x^3lnx) + 2/27x^3 +C
int x^2ln^2xdx = x^3/27(9ln^2x - 6lnx +2) +C