Constraints on fall-back discs in radio pulsars and anomalous X-ray pulsars