A route to minimally dissipative switching in magnets via THz phonon pumping