Connecting eigenvalue rigidity with polymer geometry: Diffusive transversal fluctuations under large deviation