Quantum phase transitions in mass-deformed ABJM matrix model