To Ensemble or Not Ensemble: When Does End-to-End Training Fail?