What is `d/dx(cosx(dy)/dz)` where `z=sinx`?

I assume that the problem is not with the product rule, but a question of how to express `d/dx(dy/dz)`.

Here is one way to approach the question:
The chain rule tells us that

`d/dx(g(z)) = d/dz(g(z)) * dz/dx`

In this question, `g(z) = dy/dz` so

`d/dz(g(z)) = (d^2y)/dz^2`

`d/dx(dy/dz) = d/dz(dy/dz) * dz/dx`

`= (d^2y)/dz^2 * cosx`

If our goal is to have only derivative with respect to `x`

Use the chain rule:

`dy/dx = dy/dz * dz/dx = dy/dz *cosx`

So `dy/dz = secx dy/dx`

and

`d/dx(cosxdy/dz) = d/dx(cosx(secxdy/dx)) = (d^2y)/dx^2`

Full Question:
Use `z=sinx` to transform:

`cosx(d^2y)/(dx^2)+sinxdy/dx-2ycos^3x=2cos^5x` ..... [A]

into:

`(d^2y)/(dz^2)-2y=2(1-z^2)` ..... [B]

Solution:

The aim is to eliminate `x` from the DE [A] using the suggested substitution, so as instructed:

Let `z=sinx => dz/dx = cosx`

By the chain rule:

`dy/(dx) = dy/dz * dz/dx`

`" " = cosx dy/dz` ..... [C]

Differentiating again (using the product rule)

`(d^2y)/(dx^2) = d/dx cosx dy/dz`
`" " = (cosx)( (d/dx (dy/dz)) + (d/dx (cos x) )(dy/dz)`
`" " = cosx( (dz/dx d/dz (dy/dz)) + (-sinx)(dy/dz)`
`" " = cosx (cosx (d^2y)/(dz^2) ) -sinxdy/dz`
`" " = cos^2x (d^2y)/(dz^2) -zdy/dz`

Now we substitute [C] and [D] into [A], and use `z=sinx`

`cosx(cos^2x (d^2y)/(dz^2) -zdy/dz)+z(cosx dy/dz)-2ycos^3x=2cos^5x`

`:. cos^3x (d^2y)/(dz^2) -2ycos^3x=2cos^5x`

We can cancel a factor of `cos^3x`

`:. (d^2y)/(dz^2) -2y=2cos^2x`

And using the identity `cos^2 A + sin^A -= 1` we have:

`cos^2x = 1-sin^2x = 1-z^2`

And so:

`(d^2y)/(dz^2) -2y=2(1-z^2) \ \ \ `, which is equation [B] QED

Solving the Modified DE

And, now we must solve the DE

`(d^2y)/(dz^2) -2y=2(1-z^2)`

This is a second order linear non-Homogeneous Differentiation Equation with constant coefficients. The standard approach is to find a solution, `y_c` of the homogeneous equation by looking at the Auxiliary Equation, which is the quadratic equation with the coefficients of the derivatives, and then finding an independent particular solution, `y_p` of the non-homogeneous equation.

Complementary Function

The homogeneous equation associated with [B] is

`y''-2y = 0`

And it's associated Auxiliary equation is:

`m^2-2 = 0`

Which has two real and distinct solution `m=+-sqrt(2)`

Thus the solution of the homogeneous equation is:

`y_c = Ae^(-sqrt(2)z) + Be^(+sqrt(2)z)`
`\ \ \ = Ae^(-sqrt(2)z) + Be^(sqrt(2)z)`

Particular Solution

In order to find a particular solution of the non-homogeneous equation we would look for a quadratic form, ie a solution of the form:

`y = az^2+bz+c`

Where the constants `a,b,c` are to be determined by direct substitution and comparison:

Differentiating wrt `x` we get:

`y' = 2az+b`

Differentiating again wrt `x` we get:

`y'' = 2a`

Substituting into the DE [B] we get:
`(2a) -2(az^2+bz+c) = 2 - 2z^2`

Equating coefficients of `x^0` and `x` we get:

`z^0: 2a-2c=2`
`z^1: -2b=0`
`z^2: -2a=-2`

Solving simultaneously, we have:

`a = 1`
`b = 0`
`c = 0`

And so we form the Particular solution:

`y_p = z^2`

Which then leads to the GS of [B}

`y(z) = y_c + y_p`
`\ \ \ \ \ \ \ = Ae^(-sqrt(2)z) + Be^(sqrt(2)z) + z^2`

Wrapping It Up

Now that we have a solution to the DE [B]

`y(z) = Ae^(-sqrt(2)z) + Be^(sqrt(2)z) + z^2`

We can restore the earlier substitution `z=sinx` to get:

`y(x) = Ae^(-sqrt(2)sinx) + Be^(sqrt(2)sinx) + sin^2x`

Which is the General Solution