Diversity in multiple classifier systems

Fifteen years ago, the reader would have questioned a statement that an ensemble of classifiers is generally better than a single classifier. Now this is the prevailing opinion based on a substantial amount of theoretical and empirical evidence, and on the availability of smart training methods for classifier ensembles. It is intuitively clear that an ensemble of identical classifiers will be no better than a single member thereof. If we have ‘‘the perfect classifier’’, then no ensemble is needed. If the ensemble members are imperfect, they should be different so that at least some of them are correct where the others are wrong. We call this loosely specified property diversity, and set off to explore why and how it works for the success of the ensemble, if at all. Diversity does work! Classifier ensembles that enforce diversity fare better than ones that do not. The classical example is boosting versus bagging, the two currently most successful ensemble strategies. Both approaches build the ensembles by training each classifier on a bespoke data set. Boosting promotes diversity actively whereas bagging relies on independent re-sampling from the training set. Boosting has been crowned as the ‘‘best off-the-shelf classifier’’ by Leo Breiman himself, the creator of bagging. Numerous theoretical studies explain the success of Boosting by proving bounds and margins on its error. The secret lies with the ingenious construction of the subsequent training data sets so that classifiers trained on them form a diverse ensemble. Can we not measure and use diversity explicitly to create better ensembles? Our previous studies led us to the somewhat surprising and discouraging conjecture that diversity is not unequivocally related to the ensemble accuracy. Is this a fault of defining and measuring diversity? Should diversity be always related to accuracy? Should diversity be perceived as a property of the set of classifiers or should it be related to the combination method too? This special issue, consisting of seven original contributions, looks into diversity through a magnifying glass. The efforts of leading researchers and teams are being presented together in search of answers to some of the above questions.