On the non-detection of γ-rays from energetic millisecond pulsars – dependence on viewing geometry