We will searching for stationary points, qualifying then as local maxima/minima.
First we will transform the maxima/minima with inequality restrictions into an equivalent maxima/minima problem but now with equality restrictions.
To do that we will introduce the so called slack variables s_1s1 and s_2s2 such that the problem will read.
Maximize/minimize f(x,y) = (x - y) (x + y) + sqrt[x y]f(x,y)=(x−y)(x+y)+√xy
constrained to
{
(g_1(x,y,s_1)=x y - y^2 - s_1^2=0),
(g_2(x,y,s_2)=x y - y^2 + s_2^2 - 5=0)
:}
The lagrangian is given by
L(x,y,s_1,s_2,lambda_1,lambda_2) = f(x,y)+lambda_1 g_1(x,y,s_1)+lambda_2g_2(x,y,s_2)
The condition for stationary points is
grad L(x,y,s_1,s_2,lambda_1,lambda_2)=vec 0
so we get the conditions
{
(2 x + lambda_1 y + lambda_2 y + y/(2 sqrt[x y]) = 0),
(lambda_1 (x - 2 y) + llambda_2 (x - 2 y) - 2 y + x/(2 sqrt[x y]) = 0),
(-s_1^2 + x y - y^2 = 0),
(-2 lambda_1 s_1 = 0),
(-5 + s2^2 + x y - y^2 = 0),
(2 lambda_2 s_2 = 0)
:}
Solving for {x,y,s_1,s_2,lambda_1,lambda_2} we have
{(x= -5.15905, y= -3.86559, lambda_1 = 0., s_1 = -2.23607, lambda_2= -2.78118, s_2= 0.), (x = 5.15905, y = 3.86559, lambda_1= 0., s_1 = -2.23607, lambda_2= -2.78118, s_2= 0.)
:}
The restriction g_2(x,y) is active (lambda_2 ne 0 and s_2=0) so for qualifying the maxima/minima we produce
f_{g_2}(x) = 1/2 (10 + sqrt[2] sqrt[x (x - sqrt[-20 + x^2])]
pm x (x + sqrt[x^2-20]))
Computing
d/(dx)(f_{g_2}( pm5.15905)) = 0
and
d^2/(dx^2)(f_{g_2}( pm5.15905)) = 1.64412
we conclude that the found solutions are local minima points.