Impurities or a neutral Fermi surface? A further examination of the low-energy ac optical conductivity of SmB6