How do you write two functions for which #(f*g)(x)=2x^2+11x-6#?

The trivial response is to define #g(x) = x# and #f(x)=2x^2+11x-6# then

#(f@g)(x)=2x^2+11x-6#

You can also proceed in a more general way making

#g(x) = a x + b# and
#f(x) = x^2+x + c# and then

#(f@g)(x)=a^2x^2+a(1+2b)x+b+b^2+c# and then making an equivalence of coefficients

#{(2 - a^2=0),(11 - a(1 + 2 b)=0),(6 + b + b^2 + c=0):}#

and solving for #a,b,c# giving

#a = sqrt[2], b = 1/4 (11 sqrt[2]-2), c = -167/8# or

#g(x)= sqrt[2] x+1/4 (11 sqrt[2]-2) #
#f(x)=x^2+x-167/8#

Another interesting way we could think about this is to complete the square.

#2x^2+11x-6=2(x^2+11/2x)-6#

#color(white)(2x^2+11x-6)=2(x^2+11/2x+121/16)-6-121/8#

#color(white)(2x^2+11x-6)=2(x+11/4)^2-169/8#

So, we can see that our "inner function" is #x+11/4# and the outer function would be #2x^2-169/8#.

So if #f(x)=2x^2-169/8# and #g(x)=x+11/4# then #(f@g)(x)=2x^2+11x-6#.

We can come up with another by just simplifying the function we already had.

#2(x+11/4)^2-169/8=2((4x+11)/4)^2-169/8#

#color(white)(2(x+11/4)^2-169/8)=1/8(4x+11)^2-169/8#

So we can write that for #f(x)=1/8x^2-169/8=(x^2-169)/8# and #g(x)=4x+11#, then #(f@g)(x)=2x^2+11x-6#.

Another method would be to factor first.

#2x^2+11x-6=2x^2+12x-x-6#

#color(white)(2x^2+11x-6)=(2x-1)(x+6)#

Choose some arbitrary factor, for example, #x+2#. We can express each individual factor in terms of #x+2#.

#color(white)(2x^2+11x-6)=(2(x+2)-5)((x+2)+4)#

If we let #x+2=u# this becomes

#color(white)(2x^2+11x-6)=(2u-5)(u+4)#

#color(white)(2x^2+11x-6)=2u^2+3u-20#

So we can say that if #f(x)=2x^2+3x-20# and #g(x)=x+2#, then #(f@g)(x)=2x^2+11x-6#.