Here's an incomplete list of pointers:
(1) Do all the terms have a common factor?
If so, separate out the common factor first...
#2x^2+8x+8 = 2(x^2+4x+4)#
#12x^2y - 4xy^2 + 8xy = 4xy(3x-y+2)#
(2) Do all the coefficients add up to #0#?
e.g. #f(x) = x^2+3x-4#
Here #f(1) = 1+3-4 = 0#. Hence #(x-1)# is a factor of #f(x)#
(3) Do the coefficients of the even degree terms minus the coefficients of the odd degree terms add up to #0#?
e.g. #f(x) = x^2-3x-4#
Here #f(-1) = 1-(-3)-4 = 0#. Hence #(x+1)# is a factor of #f(x)#
(4) Is the polynomial a difference or sum of squares or cubes?
#a^2 - b^2 = (a-b)(a+b)#
e.g. #4x^2-9 = (2x-3)(2x+3)#
#a^3 - b^3 = (a-b)(a^2+ab+b^2)#
e.g. #8x^6-27 = (2x^2-3)(4x^4+6x^2+9)#
#a^3 + b^3 = (a+b)(a^2-ab+b^2)#
Note that #a^2 + b^2# is only reducible to linear factors with complex coefficients.
#a^2 + b^2 = (a-ib)(a+ib)#
(5) Is the polynomial recognisable as a perfect square trinomial?
#(a+b)^2 = a^2+2ab+b^2#
#(a-b)^2 = a^2-2ab+b^2#
e.g. #4x^2-12x+9 = (2x-3)^2#
(6) Is the polynomial easily factorisable as #(x+a)(x-b)# or similar?
#(x+a)(x-b) = x^2+(a-b)x-ab#
e.g. #x^2+x-12 = (x+4)(x-3)# based on the observation that #4*3 = 12# and #4-3 = 1#
(7) Check the discriminant for a quadratic
For a quadratic in the form #f(x) = ax^2+bx+c#, check the value of the discriminant #Delta = b^2-4ac#.
If #Delta > 0# and is a perfect square, then it may be simplest to use some version of the AC Method or similar to factor the quadratic as it does have rational factors.
If #Delta > 0# is not a perfect square, then use the quadratic formula to find the roots #x_1# and #x_2# of #f(x) = 0# and hence factor #f(x)# as #a(x-x_1)(x-x_2)#
If #Delta = 0#, you have a perfect square trinomial. Look at (5) above.
If #Delta < 0#, #f(x)# has no linear factors with real coefficients. If you allow complex coefficients you can still use the quadratic formula, etc, but the result will be linear factors with complex coefficients.
