Gamma-Rays from Star Forming Activity Appear to Outshine Misaligned Active Galactic Nuclei