Optimal Ensembles for Deep Learning Classification: Theory and Practice

Ensemble methods for classification problems construct a set of models, often called "learners", and then assign class labels to new data points by taking a combination of the predictions from these models. Ensemble methods are popular and used in a wide range of problem domains because of their good performance. However, a theoretical understanding of the optimality of ensembles is, in many instances, an open problem. In particular, improving the performance of an ensemble requires an understanding of the subtle interplay between the accuracy of the individual learners and the diversity of the learners in the ensemble. For example, if all of the learners in an ensemble were identical, then clearly the accuracy of the ensemble cannot be any better than the accuracy of the individual learning, no matter how many learners one were to use. Accordingly, here we develop a theory for understanding when ensembles are optimal, in an appropriate sense, by balancing individual accuracy against ensemble diversity, from the perspective of statistical correlations. The theory that we derive is applicable for many practical ensembles, and we provide a set of metrics for assessing the optimality of any given ensemble. Perhaps most interestingly, the metrics that we develop lead naturally to a set of novel loss functions that can be optimized using backpropagation giving rise to optimal deep neural network based ensembles. We demonstrate the effectiveness of these deep neural network based ensembles using standard benchmark data sets.