Unifying neutron star sub-populations in the supernova fallback accretion model