Quantum phase transition in Bose-Fermi mixtures