I decomposed into basic fractions,
(x^2-2x)/[(x^2-3)(x-3)(x-8)]
=(Ax+B)/(x^2-3)+C/(x-3)+D/(x-8)
After expanding denominator,
(Ax+B)(x-3)(x-8)+C(x^2-3)(x-8)+D(x^2-3)(x-3)=x^2-2x
Set x=3, -30C=3, so C=-1/10
Set x=8, 305D=48, so D=48/305
Set x=0, 24B+24C+9D=0, so B=-C-3/8*D=5/122
Set x=2, 14A+14B+14C+4D=-1, so A=-7/122
Hence,
int (x^2-2x)/[(x^2-3)(x-3)(x-8)*dx]
=-1/122int ((7x-5)*dx)/(x^2-3)-1/10int (dx)/(x-3)+5/122int (dx)/(x-8)
=5/122Ln(x-8)-1/10Ln(x-3)+C-1/122*int ((7x-5)*dx)/(x^2-3)
Now I solved last integral,
int ((7x-5)*dx)/(x^2-3)
=int ((7x-5)*dx)/[(x+sqrt3)(x-sqrt3)]
=(21-5sqrt3)/6int (dx)/(x-sqrt3)-(21+5sqrt3)/6int (dx)/(x+sqrt3)
=(21-5sqrt3)/6ln(x-sqrt3)-(21+5sqrt3)/6ln(x+sqrt3)
Thus,
int (x^2-2x)/[(x^2-3)(x-3)(x-8)*dx]
=5/122Ln(x-8)+(21+5sqrt3)/732ln(x+sqrt3)-1/10Ln(x-3)-(21-5sqrt3)/6ln(x-sqrt3)+C