Under the FIRElight: Stellar Tracers of the Local Dark Matter Velocity Distribution in the Milky Way