How do you prove #cos (pi/7) cos ((2pi)/7) cos ((3pi)/7) = 1/8#?

First you should prove the following:

#\cos(2\pi/7)+\cos(4\pi/7)+\cos(6\pi/7)=-1/2#

To do that, draw a regular heptagon ABCDEFG with unit sides. Think of the sides as a vector, from A to B, then from B to C, etc. The vectors form a cycle, so from the "head to tail" rule the resultant is zero. At the same time we can define an "x-axis" from A to B and work out the components of each side's vector along that axis. Add them up and the sum must match the zero resultant:

#1+\cos(2\pi/7)+\cos(4\pi/7)+\cos(6\pi/7)+\cos(8\pi/7)+\cos(10\pi/7)+\cos(12\pi/7)=0#

Since #\cos(2\pi-x)=\cos(x)# the sum above becomes:

#1+2\cos(2\pi/7)+2\cos(4\pi/7)+2\cos(6\pi/7)=0#

From that we then have the required result

#\cos(2\pi/7)+\cos(4\pi/7)+\cos(6\pi/7)=-1/2#

With that result proven, we are now ready to tackle the given product. First use the sum-product relation for cosines

#\cos(a)\cos(b)=(1/2)(\cos(a+b)+\cos(a-b))#

to get

#\cos(\pi/7)\cos(2\pi/7)=(1/2)(\cos(3\pi/7)+\cos(-\pi/7))#

Then:

#\cos(\pi/7)\cos(2\pi/7)\cos(3\pi/7) =(1/2)(\cos(3\pi/7)(\cos(3\pi/7)+(1/2)(\cos(-\pi/7)(\cos(3\pi/7))#

Apply the sum-product relation once more to each of the above terms to get

#(1/4)(\cos(6\pi/7)+\cos(0)+\cos(2\pi/7)+\cos((-4)\pi/7))#

Since

#\cos(0)=1# and #\cos(-4\pi/7)=\cos(4\pi/7)#
this can be rearranged to

#(1/4)(1+\cos(2\pi/7)+\cos(4\pi/7)+\cos(6\pi/7))=(1/4)(1-1/2)=1/8#

we shall use the formula

#2sintheta costheta=sin2theta#

#LHS=cos(pi/7)cos((2pi)/7)cos((3pi)/7)#

#=4/(8sin(pi/7))xx2sin(pi/7)cos(pi/7)cos((2pi)/7)cos((3pi)/7)#

#=2/(8sin(pi/7))xx2sin((2pi)/7)cos((2pi)/7)cos((3pi)/7)#

#=1/(8sin(pi/7))xx2sin((4pi)/7)cos((3pi)/7)#

#=1/(8sin(pi/7))xx2sin(pi-(3pi)/7)cos((3pi)/7)#

#=1/(8sin(pi/7))xx2sin((3pi)/7)cos((3pi)/7)#

#=1/(8sin(pi/7))xxsin((6pi)/7)#

#=1/(8sin(pi/7))xxsin(pi-(6pi)/7)#

#=1/(8sin(pi/7))xxsin(pi/7)#

#=1/8#

proved