Improving deep forest by ensemble pruning based on feature vectorization and quantum walks

Recently, a deep learning model, the deep forest (DF), was designed as an alternative to deep neural networks. Each cascade layer of the DF contains a set of random forests (RFs) with a large number of decision trees, some of which are of high redundancy and poor performance. To avoid the negative impacts of such decision trees, this paper proposes to optimize RFs in each cascade layer of the DF so as to realize a pruned deep forest (PDF) with higher performance and smaller ensemble size. In this paper, a new ordering-based ensemble pruning method is proposed based on feature vectorization and quantum walks. This method simultaneously considers the accuracy and the diversity of base classifiers, and it provides an integrated evaluation criterion for ordering base classifiers in the ensemble system. The effectiveness of the proposed method is verified by experiments and discussions.

[1]  Lev V. Utkin,et al.  A deep forest classifier with weights of class probability distribution subsets , 2019, Knowl. Based Syst..

[2]  Li Bai,et al.  Cosine Similarity Metric Learning for Face Verification , 2010, ACCV.

[3]  Harris Drucker,et al.  Improving Regressors using Boosting Techniques , 1997, ICML.

[4]  Zhi-Hua Zhou,et al.  Improving Deep Forest by Confidence Screening , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[5]  Ji Feng,et al.  Deep forest , 2017, IJCAI.

[6]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[7]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[8]  Ji Feng,et al.  Distributed Deep Forest and its Application to Automatic Detection of Cash-Out Fraud , 2018, ACM Trans. Intell. Syst. Technol..

[9]  Edwin R. Hancock,et al.  Graph matching using the interference of continuous-time quantum walks , 2009, Pattern Recognit..

[10]  Salvador Elías Venegas-Andraca,et al.  Quantum walks: a comprehensive review , 2012, Quantum Information Processing.

[11]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[12]  Zhi-Hua Zhou,et al.  On the doubt about margin explanation of boosting , 2010, Artif. Intell..

[13]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[14]  Ji Feng,et al.  Deep Forest: Towards An Alternative to Deep Neural Networks , 2017, IJCAI.

[15]  Aharonov,et al.  Quantum random walks. , 1993, Physical review. A, Atomic, molecular, and optical physics.

[16]  S. Banerjee,et al.  Targeted Next Generation Sequencing Revealed a Novel Homozygous Loss-of-Function Mutation in ILDR1 Gene Causes Autosomal Recessive Nonsyndromic Sensorineural Hearing Loss in a Chinese Family , 2019, Front. Genet..

[17]  Yangming Li,et al.  An Improved Deep Forest Model for Predicting Self-Interacting Proteins From Protein Sequence Using Wavelet Transformation , 2019, Front. Genet..

[18]  Egon L. van den Broek,et al.  Fast Exact Euclidean Distance (FEED): A New Class of Adaptable Distance Transforms , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Li Wen,et al.  Rotation-Based Deep Forest for Hyperspectral Imagery Classification , 2019, IEEE Geoscience and Remote Sensing Letters.

[20]  Feng Yang,et al.  Ship Detection From Thermal Remote Sensing Imagery Through Region-Based Deep Forest , 2018, IEEE Geoscience and Remote Sensing Letters.

[21]  Mingliang Xu,et al.  Margin & diversity based ordering ensemble pruning , 2018, Neurocomputing.

[22]  Rui Ye,et al.  Considering diversity and accuracy simultaneously for ensemble pruning , 2017, Appl. Soft Comput..

[23]  Jing Xu,et al.  A Novel Deep Flexible Neural Forest Model for Classification of Cancer Subtypes Based on Gene Expression Data , 2019, IEEE Access.

[24]  Lev V. Utkin,et al.  Deep Forest as a framework for a new class of machine-learning models , 2019 .

[25]  Andris Ambainis,et al.  Quantum walks on graphs , 2000, STOC '01.

[26]  Cathy H. Wu,et al.  Neural networks for full-scale protein sequence classification: Sequence encoding with singular value decomposition , 1995, Machine Learning.

[27]  Dongdong Chen,et al.  Quantum-based subgraph convolutional neural networks , 2019, Pattern Recognit..

[28]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[29]  Lev V. Utkin,et al.  A Siamese Deep Forest , 2017, Knowl. Based Syst..

[30]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[31]  Jesús Ariel Carrasco-Ochoa,et al.  PBC4cip: A new contrast pattern-based classifier for class imbalance problems , 2017, Knowl. Based Syst..