Plasmonic dynamics measured with frequency-comb-referenced phase spectroscopy