Extraction of spin-averaged rovibrational transition frequencies in HD$^+$ for the determination of fundamental constants