With the rotation
R_(theta)=((ctheta,-stheta),(stheta,ctheta)), calling p = ((x),(y)) and P=((X),(Y)) we have first
p^T M p + 10 = 0 representing the conic, with M = ((2,-3/2),(-3/2,-2)). Here .^T represents transposition. Now introducing a change of coordinates
R_(theta)P = p we will have
P^TR_(theta)^TMR_(theta)P + 10 = 0
Calling M_(theta) = R_(theta)^TMR_(theta)=((2 Cos(2 theta) - 3 Cos(theta) Sin(theta), -3/2Cos(2 theta) -
2 Sin(2 theta)),(-3/2 Cos(2 theta) - 2 Sin(2 theta), -2 Cos(2 theta) +
3 Cos(theta) Sin(theta)))
Now, choosing theta such that -3/2 Cos(2 theta) - 2 Sin(2 theta)=0 we obtain
tan(2theta)=-3/4 and
theta = 1/2(-0.643501pmkpi) Choosing for k=0 we have
theta =-0.321751 and
M_(theta) = ((5/2,0),(0,-5/2)) and in the new coordinates, the conic reads
5/2X^2-5/2Y^2+10=0 or
X^2-Y^2+4=0 which is a hyperbola.