Spin jam induced by quantum fluctuations in a frustrated magnet