Roton-Type Mode Softening in a Quantum Gas with Cavity-Mediated Long-Range Interactions