The boundary conditions are set to be

\(\displaystyle \psi(0) = 0\)

\(\displaystyle \psi(a) = 0\)

Therefore, following substitution for \(\displaystyle \psi\), you get

\(\displaystyle A=0\)

and

\(\displaystyle B\sin \omega a = 0\)

You could just set B = 0 as well, but that doesn't yield a very interesting solution. That just states that when the wave function is 0, you satisfy the potentials. Therefore, the more interesting solution is when you look at the fluctuating sine wave and compare the results with 0. It turns out that the LHS is equal to 0 when

\(\displaystyle \omega a = \pi n\)

where n is any integer. If the above is true, B can be anything and it will still be valid.

You can verify relationship by picking an integer (say, n=2) and then plot the sine wave. You'll see that no matter what you choose, the curve will drops down to x=0 at the boundary. The only exception is if you pick n=0 solution, which gives the same result as the B=0 solution.

Another constraint is required to pin down what B is. In this case, it is found using normalisation (since the area under the probability density function, which is the square of the wave function, must be 1). The relationship \(\displaystyle \omega a = \pi n\) is useful when characterising the energy, E, which must have certain discrete values. This is the key feature of quantum mechanics; a lot of the solutions to problems are discrete (i.e. have an integer "n" in it somewhere)