Multi-View Forest: A New Ensemble Method based on Dempster-Shafer Evidence Theory

This paper proposes a new ensemble method that constructs an ensemble of tree-structured classifiers using multi-view learning. We are motivated by the fact that an ensemble can outperform its members providing that these classifiers are diverse and accurate. In order to construct diverse individual classifiers, we assume that the object to be classified is described by multiple feature sets (views). We aim to construct different tree classifiers using different views to improve the accuracy of the multi-class recognition task. For the decision fusion of the binary classifiers within each tree classifier, Dempster’s unnormalized rule of combination is applied and an evidence-theoretic decision profile is proposed to combine the decisions of different trees. Numerical experiments have been performed on two real-world data sets: a data set of handwritten digits, and another data set of 3D visual objects. The results indicate that the proposed forest efficiently integrates multi-view data and outperforms the individual tree classifiers.

[1]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[2]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[3]  Günther Palm,et al.  Comparison of Multiclass SVM Decomposition Schemes for Visual Object Recognition , 2005, DAGM-Symposium.

[4]  Rayid Ghani,et al.  Combining labeled and unlabeled data for text classification with a large number of categories , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[5]  Sanjoy Dasgupta,et al.  Hybrid Hierarchical Clustering: Forming a Tree From Multiple Views , 2005 .

[6]  Goo Jun,et al.  Multi-class Boosting with Class Hierarchies , 2009, MCS.

[7]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[8]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[9]  Günther Palm,et al.  Semi-supervised learning for tree-structured ensembles of RBF networks with Co-Training , 2010, Neural Networks.

[10]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[11]  Goo Jun,et al.  Hybrid Hierarchical Classifiers for Hyperspectral Data Analysis , 2009, MCS.

[12]  Joydeep Ghosh,et al.  Modular learning through output space decomposition , 2000 .

[13]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[14]  Friedhelm Schwenker,et al.  Decision Templates Based RBF Network for Tree-Structured Multiple Classifier Fusion , 2009, MCS.

[15]  Joydeep Ghosh,et al.  An Empirical Comparison of Hierarchical vs. Two-Level Approaches to Multiclass Problems , 2004, Multiple Classifier Systems.

[16]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[17]  Günther Palm,et al.  Tree-Structured Support Vector Machines for Multi-class Pattern Recognition , 2001, Multiple Classifier Systems.

[18]  Kari Sentz,et al.  Combination of Evidence in Dempster-Shafer Theory , 2002 .

[19]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[20]  Rebecca Fay,et al.  Feature selection and information fusion in hierarchical neural networks for iterative 3D-object recognition , 2007 .

[21]  Robert Tibshirani,et al.  Margin Trees for High-dimensional Classification , 2007, J. Mach. Learn. Res..

[22]  Friedhelm Schwenker,et al.  Three learning phases for radial-basis-function networks , 2001, Neural Networks.

[23]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[24]  Galina L. Rogova,et al.  Combining the results of several neural network classifiers , 1994, Neural Networks.