Bistability and instability of three-level atoms inside an optical cavity