Probing the limits of optical cycling in a predissociative diatomic molecule