Start with the identity:
#tan^2(alpha) +1 = sec^2(alpha)#
Substitute #sec^2(alpha)= 1/cos^2(alpha)#
#tan^2(alpha)+ 1= 1/cos^2(alpha)#
Multiply both sides by #cos^2(alpha)/(tan^2(alpha) + 1)#:
#cos^2(alpha) = 1/(tan^2(alpha) + 1)#
Use the square root operation on both sides:
#cos(alpha) = +-sqrt(1/(tan^2(alpha) + 1))#
We are told that #180^@ < alpha < 270^@#, therefore we choose the negative value:
#cos(alpha) = -sqrt(1/(tan^2(alpha) + 1))#
Add 1 to both sides:
#1+ cos(alpha) = 1-sqrt(1/(tan^2(alpha) + 1))#
Multiply both sides by #1/2#:
#(1+ cos(alpha))/2 = (1-sqrt(1/(tan^2(alpha) + 1)))/2#
Use the square root operation on both sides:
#+-sqrt((1+ cos(alpha))/2) = +-sqrt((1-sqrt(1/(tan^2(alpha) + 1)))/2)#
Substitute #+-sqrt((1+ cos(alpha))/2)= cos(alpha/2)#
#cos(alpha/2) = +-sqrt((1-sqrt(1/(tan^2(alpha) + 1)))/2)#
From #180^@ < alpha < 270^@# we derive #90^@ < alpha/2 < 135^@# and conclude that the cosine function is negative within the specified domain:
#cos(alpha/2) = -sqrt((1-sqrt(1/(tan^2(alpha) + 1)))/2)#
Substitute #tan^2(alpha) = (40/9)^2#:
#cos(alpha/2) = -sqrt((1-sqrt(1/((40/9)^2 + 1)))/2)#
I used WolframAlpha to simplify the above into an exact form:
#cos(alpha/2) = -(4sqrt41)/41#