How do I find the angle of rotation of #153x^2 - 192xy + 97y^2 - 30x -40y - 200 = 0#, given #cot 2theta = (A - C)/B#?

# cot 2theta = (153-97)/(-192) = -56/192#

So #2theta = cot^-1(-56/192)# and

#theta = 1/2cot^-1(-56/192)# Get out a calculator or trig tables. That is not a special angle we know.

But wait, there's more.

Perhaps we are really interested in eliminating the #xy# term from the equation. In which case, we don't really need to know #theta#, we just need the sine and cosine of #theta#. Those we can get without trig tables or calculators.

#-56/192 = -28/96 = -7/24# AHA! (or perhaps not)
Even if we don't immediately see it, that is a #7, 24, 25# right triangle.
Or use trigonometry: if the cotangent of an angle is #-7/24#, find the cosine of that angle.

If #cot 2theta = -7/24#, the #cos 2 theta = -7/25#.

Now we can use half-angle formulas to get #sin theta# and cos theta#

And then we can do the substitution to eliminate the #xy# term.