Orientational states, phase transitions, and spectra of vibrational excitations for two-dimensional systems with quadrupole interactions