Why Optically Faint AGNs Are Optically Faint: The Spitzer Perspective