Long-lived modulation of plasmonic absorption by ballistic thermal injection