Spectroscopic characterization of the quantum linear-zigzag transition in trapped ions