Modeling dark photon oscillations in our inhomogeneous Universe