Vibrational instability, two-level systems, and the boson peak in glasses