What are the set of d orbitals involved in forming capped octahedral geometry?

#d_(z^2)#, #d_(x^2-y^2)#, and #d_(xy)#

OR

#d_(z^2)#, #d_(xz)#, and #d_(yz)#

To visualize this geometry more clearly, go here and play around with the animation GUI.

A capped octahedral geometry is basically octahedral with an extra ligand in between the equatorial ligands, above the equatorial plane:

![http://www.chemtube3d.com/]()

The principal axis of rotation here is a #C_3(z)# axis, and this is in the #C_(3v)# point group. Another way to view this is down this #C_3(z)# axis:

![http://www.chemtube3d.com/]()

Since the #z# axis points through the cap atom, that's where the #d_(z^2)# points. The atoms on the octahedral face (that form the triangle in the second view) are on the #xy# plane, so we need both the on-axis and off-axis #d# orbitals (the #x^2-y^2# and #xy#) to describe this hybridization.

Therefore, one option I would guess is #z^2, x^2-y^2, xy#.

If you're into group theory, the character table for #C_(3v)# is:

![)

The reducible representation is gotten by operating with #hatE#, #hatC_3#, and #hatsigma_v#; I chose an #s# orbital basis, so that unmoved atoms return a #1#, and moved atoms return a #0#.

This turns out to be:

#" "" "hatE" "2hatC_3" "3hatsigma_v#
#Gamma_(sigma) = 7" "1" "" "3#

and this reduces down to:

#Gamma_(sigma)^(red) = 3A_1 + 2E#

On the character table,

• #s harr x^2 + y^2#
• #p_x harr x#
• #p_y harr y#
• #p_z harr z#
• #d_(z^2) harr z^2#
• #d_(x^2-y^2) harr x^2-y^2#
• #d_(xy) harr xy#
• #d_(xz) harr xz#
• #d_(yz) harr yz#

Therefore, this can correspond to the linear combination:

#overbrace(s)^(A_1) + overbrace(p_z)^(A_1) + overbrace(d_(z^2))^(A_1) + overbrace((p_x", "p_y))^(E) + overbrace((d_(x^2-y^2)", "d_(xy)))^(E)#

#ul("orbital"" "" "" ""IRREP")#
#s" "" "" "" "" "" "A_1#
#p_z" "" "" "" "" "color(white)(.)A_1#
#(p_x,p_y)" "" "" "color(white)(.)E#
#d_(z^2)" "" "" "" "color(white)(....)A_1#
#(d_(x^2-y^2), d_(xy))" "color(white)(.)E#

The other choice, though not as easy to see, is:

#overbrace(s)^(A_1) + overbrace(p_z)^(A_1) + overbrace(d_(z^2))^(A_1) + overbrace((p_x", "p_y))^(E) + overbrace((d_(xz)", "d_(yz)))^(E)#

#ul("orbital"" "" "" ""IRREP")#
#s" "" "" "" "" "" "A_1#
#p_z" "" "" "" "" "color(white)(.)A_1#
#(p_x,p_y)" "" "" "color(white)(.)E#
#d_(z^2)" "" "" "" "color(white)(....)A_1#
#(d_(xz), d_(yz))" "" "color(white)(..)E#