Here,
#Cot^2x = -2cotx -1 #
#=>cot^2x+2cotx+1=0#
#=>(cotx+1)^2=0#
#=>cotx+1=0#
#color(red)(=>cotx=-1...(A)#
#=>tanx=-1...# #tocosx!=0#
#=>tanx=tan(-pi/4)#
#color(blue)(=>x=kpi-pi/4,kinZZ...to(B)#
We know that ,the range of #cot^-1x, is :(0.pi)#
Now from #(A)#
#cotx=-1=>x=cot^-1(-1)!=-pi/4...toIV^(th)Quadrant#
#and-pi/4!in(0,pi)#
So, #color(red)(cot^-1(-1)=pi-pi/4=(3pi)/4...toII^(nd)Quadrant#
Hence, from #(B)#
#x=(2k+1)pi-pi/4,kinZZ#
Note:
#color(blue)(x={color(red)(kpi)-pi/4,kinZZ}#
#:.x={color(red)((2k+1)pi)-pi/4,kinZZ}uu{color(red)((2k)pi)-
pi/4,kinZZ}#
#color(white)(.................)color(red)(II^(nd)Quadrant)color(white)
(...................)color(red)(IV^(th)Quadrant)#