Đáp án đúng là: A

Xét tứ diện

\(ABCD\) cạnh \(a\) ta có: \(\overrightarrow {DM} = \frac{{a\sqrt 3 }}{2};AM = \frac{{a\sqrt 3 }}{2}\).

Ta có: \(\overrightarrow {AB} .\overrightarrow {DM} = \overrightarrow {AB} .\left( {\overrightarrow {AM} - \overrightarrow {AD} } \right) = \overrightarrow {AB} .\overrightarrow {AM} - \overrightarrow {AB} .\overrightarrow {AD} \)

\( = \left| {\overrightarrow {AB} } \right|.\left| {\overrightarrow {AM} } \right|.\cos \left( {\overrightarrow {AB} ,\overrightarrow {AM} } \right) - \left| {\overrightarrow {AB} } \right|.\left| {\overrightarrow {AD} } \right|.\cos \left( {\overrightarrow {AB} ,\overrightarrow {AD} } \right)\)

= \(a.\frac{{a\sqrt 3 }}{2}.\frac{{\sqrt 3 }}{2} - a.a.\frac{1}{2} = \frac{{{a^2}}}{4}\).

Ta có: \(\cos \left( {\overrightarrow {AB} ,\overrightarrow {DM} } \right)\) = \(\frac{{\overrightarrow {AB} .\overrightarrow {DM} }}{{\left| {\overrightarrow {AB} } \right|.\left| {\overrightarrow {DM} } \right|}} = \frac{{\frac{{{a^2}}}{4}}}{{a.\frac{{a\sqrt 3 }}{2}}} = \frac{{\sqrt 3 }}{6}\).