Near-infrared spectroscopy of post-starburst galaxies: a limited impact of TP-AGB stars on galaxy spectral energy distributions