Classification and regression trees

Classification and regression trees are machine‐learning methods for constructing prediction models from data. The models are obtained by recursively partitioning the data space and fitting a simple prediction model within each partition. As a result, the partitioning can be represented graphically as a decision tree. Classification trees are designed for dependent variables that take a finite number of unordered values, with prediction error measured in terms of misclassification cost. Regression trees are for dependent variables that take continuous or ordered discrete values, with prediction error typically measured by the squared difference between the observed and predicted values. This article gives an introduction to the subject by reviewing some widely available algorithms and comparing their capabilities, strengths, and weakness in two examples. © 2011 John Wiley & Sons, Inc. WIREs Data Mining Knowl Discov 2011 1 14‐23 DOI: 10.1002/widm.8

[1]  E. B. Wilson,et al.  The Distribution of Chi-Square. , 1931, Proceedings of the National Academy of Sciences of the United States of America.

[2]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[3]  F. E. Satterthwaite.  An approximate distribution of estimates of variance components. , 1946, Biometrics.

[4]  J. Morgan,et al.  Problems in the Analysis of Survey Data, and a Proposal , 1963 .

[5]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[6]  R. C. Messenger,et al.  A Modal Search Technique for Predictive Nominal Scale Multivariate Analysis , 1972 .

[7]  A. Fielding,et al.  Binary Segmentation in Survey Analysis with Particular Reference to AID , 1977 .

[8]  W. W. Muir,et al.  Regression Diagnostics: Identifying Influential Data and Sources of Collinearity , 1980 .

[9]  G. V. Kass.  An Exploratory Technique for Investigating Large Quantities of Categorical Data , 1980 .

[10]  G. Grisetti,et al.  Further Reading , 1984, IEEE Spectrum.

[11]  Bernard R. Rosner,et al.  Fundamentals of Biostatistics. , 1992 .

[12]  W. Loh,et al.  Tree-Structured Classification via Generalized Discriminant Analysis. , 1988 .

[13]  Mark R. Segal,et al.  Regression Trees for Censored Data , 1988 .

[14]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[15]  James M. Landwehr,et al.  Robust materials and processes: Key to reliability , 1990, AT&T Technical Journal.

[16]  A. Ciampi.  Generalized regression trees , 1991 .

[17]  M. LeBlanc,et al.  Relative risk trees for censored survival data. , 1992, Biometrics.

[18]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[19]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[20]  Robin H. Lock.  1993 New Car Data , 1993 .

[21]  W. Loh,et al.  Tree-structured proportional hazards regression modeling. , 1994, Biometrics.

[22]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[23]  P. Chaudhuri,et al.  Piecewise polynomial regression trees , 1994 .

[24]  H. Ahn.  Tree-Structured Exponential Regression Modeling† , 1994 .

[25]  W. Loh,et al.  Generalized regression trees , 1995 .

[26]  W. Loh,et al.  SPLIT SELECTION METHODS FOR CLASSIFICATION TREES , 1997 .

[27]  H. Chipman,et al.  Bayesian CART Model Search , 1998 .

[28]  Adrian F. M. Smith,et al.  A Bayesian CART algorithm , 1998 .

[29]  J. Skolnick,et al.  Application of an artificial neural network to predict specific class I MHC binding peptide sequences , 1998, Nature Biotechnology.

[30]  Kathryn Parker Boudett,et al.  Do Male Dropouts Benefit from Obtaining a GED, Postsecondary Education, and Training? , 1999, Evaluation review.

[31]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[32]  Ian Witten,et al.  Data Mining , 2000 .

[33]  M. Segal,et al.  Relating Amino Acid Sequence to Phenotype: Analysis of Peptide‐Binding Data , 2000, Biometrics.

[34]  Hyunjoong Kim,et al.  Classification Trees With Unbiased Multiway Splits , 2001 .

[35]  W. Loh,et al.  REGRESSION TREES WITH UNBIASED VARIABLE SELECTION AND INTERACTION DETECTION , 2002 .

[36]  W. Loh,et al.  Nonparametric estimation of conditional quantiles using quantile regression trees ∗ ( Published in Bernoulli ( 2002 ) , 8 , 561 – 576 ) , 2008 .

[37]  J. Singer,et al.  Applied Longitudinal Data Analysis , 2003 .

[38]  Hyunjoong Kim,et al.  Classification Trees With Bivariate Linear Discriminant Node Models , 2003 .

[39]  S. Lemon,et al.  Classification and regression tree analysis in public health: Methodological review and comparison with logistic regression , 2003, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[40]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[41]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[42]  Xiaogang Su,et al.  Joint Statistical Meetings- Statistical Computing Section Maximum Likelihood Regression Trees , 2022 .

[43]  W. Loh,et al.  LOTUS: An Algorithm for Building Accurate and Comprehensible Logistic Regression Trees , 2004 .

[44]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[45]  G. Molenberghs.  Applied Longitudinal Analysis , 2005 .

[46]  M. Kahn.  An Exhalent Problem for Teaching Statistics , 2005 .

[47]  K. Hornik,et al.  Unbiased Recursive Partitioning: A Conditional Inference Framework , 2006 .

[48]  W. Loh,et al.  Logistic Regression Tree Analysis , 2006 .

[49]  W. Loh,et al.  Regression tree models for designed experiments , 2006, math/0611192.

[50]  W. Loh,et al.  Logistic Regression Tree Analysis ( In Handbook of Engineering Statistics , 2006 .

[51]  Wei Zheng,et al.  Extrapolation errors in linear model trees , 2007, TKDD.

[52]  I-Cheng Yeh,et al.  Modeling slump flow of concrete using second-order regressions and artificial neural networks , 2007 .

[53]  Hyunjoong Kim,et al.  Visualizable and interpretable regression models with good prediction power , 2007 .

[54]  Hyung Jun Cho,et al.  Median Regression Tree for Analysis of Censored Survival Data , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[55]  K. Hornik,et al.  Model-Based Recursive Partitioning , 2008 .

[56]  Wei-Yin Loh,et al.  Classification and Regression Tree Methods , 2008 .

[57]  K. Doksum,et al.  Nonparametric Variable Selection: The EARTH Algorithm , 2008 .

[58]  Wei-Yin Loh,et al.  Regression by Parts: Fitting Visually Interpretable Models with GUIDE , 2008 .

[59]  G. Tutz,et al.  An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. , 2009, Psychological methods.

[60]  W. Loh,et al.  Improving the precision of classification trees , 2010, 1011.0608.

[61]  Victoria A. Shaffer,et al.  Binary recursive partitioning: background, methods, and application to psychology. , 2011, The British journal of mathematical and statistical psychology.

[62]  R. Olshen,et al.  Points of Significance: Classification and regression trees , 2017, Nature Methods.

[63]  W. Loh,et al.  Regression trees for longitudinal and multiresponse data , 2012, 1209.4690.