Characterizing the hydrodynamic response to the initial conditions