Quantum optomechanics in the bistable regime