The largest side of a right triangle is #a^2+b^2# and other side is #2ab#. What condition will make the third side to be the smallest side?

The longest side of a right triangle is always the hypotenuse. So we know the length of the hypotenuse is #a^2+b^2.#

Let the unknown side length be #c.# Then from the Pythagorean theorem, we know

#(2ab)^2+c^2=(a^2+b^2)^2#
or
#c=sqrt((a^2+b^2)^2-(2ab)^2)#
#color(white)c=sqrt(a^4+2a^2b^2+b^4-4a^2b^2)#
#color(white)c=sqrt(a^4-2a^2b^2+b^4)#
#color(white)c=sqrt((a^2-b^2)^2)#
#color(white)c=a^2-b^2#

We also require that all side lengths be positive, so

• #a^2+b^2>0#
  #=>a!=0 or b!=0#
• #2ab>0#
  #=>a,b>0 or a,b<0#
• #c=a^2-b^2>0#
  #<=>a^2>b^2#
  #<=>absa>absb#

Now, for any triangle, the longest side must be shorter than the sum of the other two sides. So we have:

#color(white)(=>)2ab+"       "c color(white)(XX)>a^2+b^2#
#=>2ab+(a^2-b^2)>a^2+b^2#
#=>2ab color(white)(XXXXXX)>2b^2#

#=>{(a>b"," if b > 0), ( a < b"," if b < 0 ):}#

Further, for third side to be smallest, #a^2-b^2 < 2ab#
or #a^2-2ab+b^2 < 2b^2# or #a-b < sqrt2b# or #a < b(1+sqrt2)#

Combining all of these restrictions, we can deduce that in order for the third side to be the shortest, we must have #(1+sqrt2)|b|>absa>absb and (a,b<0 or a,b>0).#