What is #d/dx(cosx(dy)/dz)# where #z=sinx#?

I assume that the problem is not with the product rule, but a question of how to express #d/dx(dy/dz)#.

Here is one way to approach the question:
The chain rule tells us that

#d/dx(g(z)) = d/dz(g(z)) * dz/dx#

In this question, #g(z) = dy/dz# so

#d/dz(g(z)) = (d^2y)/dz^2#

#d/dx(dy/dz) = d/dz(dy/dz) * dz/dx#

# = (d^2y)/dz^2 * cosx#

If our goal is to have only derivative with respect to #x#

Use the chain rule:

#dy/dx = dy/dz * dz/dx = dy/dz *cosx#

So #dy/dz = secx dy/dx#

and

#d/dx(cosxdy/dz) = d/dx(cosx(secxdy/dx)) = (d^2y)/dx^2#

Full Question:
Use #z=sinx# to transform:

# cosx(d^2y)/(dx^2)+sinxdy/dx-2ycos^3x=2cos^5x# ..... [A]

into:

# (d^2y)/(dz^2)-2y=2(1-z^2)# ..... [B]

Solution:

The aim is to eliminate #x# from the DE [A] using the suggested substitution, so as instructed:

Let #z=sinx => dz/dx = cosx#

By the chain rule:

# dy/(dx) = dy/dz * dz/dx #

# " " = cosx dy/dz # ..... [C]

Differentiating again (using the product rule)

# (d^2y)/(dx^2) = d/dx cosx dy/dz #
# " " = (cosx)( (d/dx (dy/dz)) + (d/dx (cos x) )(dy/dz)#
# " " = cosx( (dz/dx d/dz (dy/dz)) + (-sinx)(dy/dz)#
# " " = cosx (cosx (d^2y)/(dz^2) ) -sinxdy/dz#
# " " = cos^2x (d^2y)/(dz^2) -zdy/dz#

Now we substitute [C] and [D] into [A], and use #z=sinx#

# cosx(cos^2x (d^2y)/(dz^2) -zdy/dz)+z(cosx dy/dz)-2ycos^3x=2cos^5x#

# :. cos^3x (d^2y)/(dz^2) -2ycos^3x=2cos^5x#

We can cancel a factor of #cos^3x#

# :. (d^2y)/(dz^2) -2y=2cos^2x#

And using the identity #cos^2 A + sin^A -= 1# we have:

# cos^2x = 1-sin^2x = 1-z^2 #

And so:

# (d^2y)/(dz^2) -2y=2(1-z^2) \ \ \ #, which is equation [B] QED

Solving the Modified DE

And, now we must solve the DE

# (d^2y)/(dz^2) -2y=2(1-z^2) #

This is a second order linear non-Homogeneous Differentiation Equation with constant coefficients. The standard approach is to find a solution, #y_c# of the homogeneous equation by looking at the Auxiliary Equation, which is the quadratic equation with the coefficients of the derivatives, and then finding an independent particular solution, #y_p# of the non-homogeneous equation.

Complementary Function

The homogeneous equation associated with [B] is

# y''-2y = 0#

And it's associated Auxiliary equation is:

# m^2-2 = 0 #

Which has two real and distinct solution #m=+-sqrt(2)#

Thus the solution of the homogeneous equation is:

# y_c = Ae^(-sqrt(2)z) + Be^(+sqrt(2)z) #
# \ \ \ = Ae^(-sqrt(2)z) + Be^(sqrt(2)z) #

Particular Solution

In order to find a particular solution of the non-homogeneous equation we would look for a quadratic form, ie a solution of the form:

# y = az^2+bz+c #

Where the constants #a,b,c# are to be determined by direct substitution and comparison:

Differentiating wrt #x# we get:

# y' = 2az+b #

Differentiating again wrt #x# we get:

# y'' = 2a #

Substituting into the DE [B] we get:
# (2a) -2(az^2+bz+c) = 2 - 2z^2 #

Equating coefficients of #x^0# and #x# we get:

#z^0: 2a-2c=2#
#z^1: -2b=0 #
#z^2: -2a=-2 #

Solving simultaneously, we have:

# a = 1 #
# b = 0 #
# c = 0 #

And so we form the Particular solution:

# y_p = z^2 #

Which then leads to the GS of [B}

# y(z) = y_c + y_p #
# \ \ \ \ \ \ \ = Ae^(-sqrt(2)z) + Be^(sqrt(2)z) + z^2#

Wrapping It Up

Now that we have a solution to the DE [B]

# y(z) = Ae^(-sqrt(2)z) + Be^(sqrt(2)z) + z^2#

We can restore the earlier substitution #z=sinx# to get:

# y(x) = Ae^(-sqrt(2)sinx) + Be^(sqrt(2)sinx) + sin^2x#

Which is the General Solution