Your method can work.
#intdx/(9x^2-16)#
#3x=4sectheta# so #3dx=4secthetatanthetad theta#:
#=int(4/3secthetatanthetad theta)/(16sec^2theta-16)d theta=1/12int(secthetatantheta)/tan^2thetad theta=1/12intcscthetad theta#
#=1/12lnabs(csctheta-cottheta)#
Which can be rewritten as:
#=1/12lnabs(sectheta/tantheta-1/tantheta)=1/12lnabs((sectheta-1)/tantheta)#
Then using #sectheta=3/4x# and #tantheta=sqrt(sec^2theta-1)=sqrt(9/16x^2-1)=1/4sqrt(9x^2-16)#:
#=1/12lnabs((3/4x-1)/(1/4sqrt(9x^2-16)))#
#=1/12lnabs((3x-4)/sqrt(9x^2-16))#
This is where the "simplification" gets tricky:
#=1/12lnabs(sqrt((3x-4)^2/(9x^2-16)))#
Bringing the square root out of the logarithm as a #1//2# power using #log(a^b)=blog(a)#:
#=1/24lnabs(((3x-4)^2)/(9x^2-16))#
#=1/24lnabs((3x-4)^2/((3x+4)(3x-4)))#
#=1/24lnabs((3x-4)/(3x+4))+C#
Which is the answer found through using partial fractions. As much as I love trig substitutions, there are definitely times when using partial fractions will get you to the simplest answer the fastest.