Consider the first equation you are given:

$ | \hat A \times \hat B | = - \sqrt 3 \ \hat A \cdot \hat B $

The left hand side is a positive number, since it's a magnitude. Therefore the right hand side must also be positive, which means $ \hat A \cdot \hat B$ must be negative. This rules out the possibility that theta = -60 degrees.