Limits of Astrophysics with Gravitational-Wave Backgrounds