Considering B_1 and B_2 with B_1 at the botton, its static equilibrium is attained with the forces
For B_1
Horizontally
f_1-f_c cos theta=0
Vertically
-m g + n_1 -f_c sintheta=0
here f_1 is the horizontal contact force with the tube wall and f_c is the contact force between B_1 and B_2 and n_1 the floor contact force.
For B_2
Horizontally
f_c costheta-f_2=0
Vertically
-m g+ f_c sintheta = 0
here f_2 is the horizontal contact force with the tube ball.
Now the global equilibrium states that the momentum of the forces acting over the tube must sum zero or
f_1 r-f_2(r+2r sintheta)+M g R =0
Here we suppose that the tube has the right bottom extremum as possible rotational pivot, at the knock down limit, so the support normal forces are concentrated at this point.
Here costheta=(R-r)/r and sintheta = sqrt[2 r R - R^2]/r
Solving the system of equations
{
(f_1-f_c cos theta=0),
(-m g + n_1 -f_c sintheta=0),
(f_c costheta-f_2=0),
(-m g +f_c sintheta = 0),
(f_1 r-f_2(r+2r sintheta)+M g R =0)
:}
for f_1,f_2,f_c,n_1,M we obtain
f_1 = (costheta g m)/sintheta, f_2 = (costheta g m)/sintheta, f_c = (g m)/sintheta, n_1 =
2 g m, M = (2 costheta m r)/R
so
M = 2m(1-r/R)
d'Alembert's principle analysis will come later.
Analysing the virtual work of each body B_1,B_2 and T (for tube) we have delta w = << delta p, f >>
delta p_1= r delta alpha hat i
delta p_2 = delta p_1 + 2r(cos(theta-delta alpha)-cos(theta)) hat i + 2r(sin(theta-delta alpha)-sin(theta))hat j so
delta p_2 approx r delta alpha hat i + 2r sintheta delta alpha hat i -2rcostheta delta alpha hat j
Now calling p_T = l cos alpha hati + l sin alpha hatj where l is the distance between the pivot point and the tube center of mass, we have
delta p_T = -l sin alpha delta alpha hat i+l cos alpha delta alpha hat j
Finally
delta w = << delta p_1, -m g hat j>> + << delta p_2,-m g hat j >> + << delta p_T, -M g hat j>> = 0 or
2 g m r cos theta - g l M cos alpha = 0 solving for M
M = (2 m r cos theta)/(l cos(alpha)) but l cos(alpha)=R and costheta = (R-r)/r so finally
M = 2m(1-r/R)
Note:
The detailed calculation of deltap_2 follows. As we know
p_1 = (r-2R)hati+r hat j and
p_2=p_1+2r costheta hati+2r sintheta hatj
After a rigid rotation of delta alpha counterclockwise the p_2 new coordinates are
p'_2=p'_1+2rcos(theta-deltaalpha)hati+2rsin(theta-deltaalpha)hatj
then
deltap_2 = p'_2-p_2
but
cos(theta-deltaalpha)=costhetacosdeltaalpha+sinthetasindeltaalpha
sin(theta-deltaalpha)=sintheta cosdeltaalpha-costhetasindeltaalpha
and also
cosdeltaalpha approx 1 and sindeltaalpha approx deltaalpha
Finally
delta p_2 = r delta alpha hat i + 2r sintheta delta alpha hat i -2rcostheta delta alpha hat j