Quantum mode-coupling theory for vibrational excitations of glasses