Unstable modes in liquids density of states, potential energy, and heat capacity