$ | \hat A \times \hat B | = - \sqrt 3 \ \hat A \cdot \hat B $

The left hand side is a positive number, since it's a magnitude. Therefore the right hand side must also be positive, which means $ \hat A \cdot \hat B$ must be negative. This rules out the possibility that theta = -60 degrees.

$ | \hat A \times \hat B | = - \sqrt 3 \ \hat A \cdot \hat B $

That expression is not in the paper he posted. The cross product is defined for two vectors whose magnitude does not appear explicitly in the expression defining it;.