We present physics-based preconditioning and a time-stepping strategy for a moment-based scale-bridging algorithm applied to the thermal radiative transfer equation. Our goal is to obtain (asymptotically) second-order time accurate and consistent solutions without nonlinear iterations between the high-order (HO) transport equation and the low-order (LO) continuum system within a time step. Modified equation analysis shows that this can be achieved via a simple predictor-corrector time stepping that requires one inversion of the transport operator per time step. We propose a physics-based preconditioning based on a combination of the nonlinear elimination technique and Fleck--Cummings linearization. As a result, the LO system can be solved efficiently via a multigrid preconditioned Jacobian-free Newton--Krylov method. For a set of numerical test problems, the physics-based preconditioner reduces the number of GMRES iterations by a factor of 3$\sim$4 as compared to a standard preconditioner for advection-di...