Stability and phase transition of localized modes in Bose-Einstein condensates with both two- and three-body interactions