Random Forests and Kernel Methods

Random forests are ensemble methods which grow trees as base learners and combine their predictions by averaging. Random forests are known for their good practical performance, particularly in high-dimensional settings. On the theoretical side, several studies highlight the potentially fruitful connection between the random forests and the kernel methods. In this paper, we work out this connection in detail. In particular, we show that by slightly modifying their definition, random forests can be rewritten as kernel methods (called KeRF for kernel based on random forests) which are more interpretable and easier to analyze. Explicit expressions of KeRF estimates for some specific random forest models are given, together with upper bounds on their rate of consistency. We also show empirically that the KeRF estimates compare favourably to the random forest estimates.

[1]  Luc Devroye,et al.  Cellular Tree Classifiers , 2013, ALT.

[2]  Adele Cutler,et al.  PERT – Perfect Random Tree Ensembles , 2001 .

[3]  Jean-Philippe Vert,et al.  Consistency of Random Forests , 2014, 1405.2881.

[4]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[5]  Yanjun Qi.  Random Forest for Bioinformatics , 2012 .

[6]  Pierre Geurts,et al.  Extremely randomized trees , 2006, Machine Learning.

[7]  E. Nadaraya.  On Estimating Regression , 1964 .

[8]  S. Geer,et al.  High-dimensional additive modeling , 2008, 0806.4115.

[9]  Luc Devroye,et al.  Consistency of Random Forests and Other Averaging Classifiers , 2008, J. Mach. Learn. Res..

[10]  A. Krzyżak,et al.  Distribution-Free Pointwise Consistency of Kernel Regression Estimate , 1984 .

[11]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[12]  G. Hooker,et al.  Ensemble Trees and CLTs: Statistical Inference for Supervised Learning , 2014 .

[13]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[14]  Hans Knutsson,et al.  Reinforcement Learning Trees , 1996 .

[15]  L. Breiman.  SOME INFINITY THEORY FOR PREDICTOR ENSEMBLES , 2000 .

[16]  Udaya B. Kogalur,et al.  Consistency of Random Survival Forests. , 2008, Statistics & probability letters.

[17]  Stefan Wager.  Asymptotic Theory for Random Forests , 2014, 1405.0352.

[18]  Erwan Scornet,et al.  On the asymptotics of random forests , 2014, J. Multivar. Anal..

[19]  Misha Denil,et al.  Consistency of Online Random Forests , 2013, ICML.

[20]  Yi Lin,et al.  Random Forests and Adaptive Nearest Neighbors , 2006 .

[21]  Zoubin Ghahramani,et al.  The Random Forest Kernel and other kernels for big data from random partitions , 2014, ArXiv.

[22]  Sylvain Arlot,et al.  Analysis of purely random forests bias , 2014, ArXiv.

[23]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[24]  Luc Devroye,et al.  On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification , 2010, J. Multivar. Anal..

[25]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[26]  Robin Genuer,et al.  Random Forests: some methodological insights , 2008, 0811.3619.

[27]  M. J. van der Laan,et al.  Statistical Applications in Genetics and Molecular Biology Super Learner , 2010 .

[28]  Philip H. S. Torr,et al.  Randomized trees for human pose detection , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Gérard Biau,et al.  Analysis of a Random Forests Model , 2010, J. Mach. Learn. Res..

[30]  L. Breiman.  CONSISTENCY FOR A SIMPLE MODEL OF RANDOM FORESTS , 2004 .