Gaining insight with recursive partitioning of generalized linear models

Recursive partitioning algorithms separate a feature space into a set of disjoint rectangles. Then, usually, a constant in every partition is fitted. While this is a simple and intuitive approach, it may still lack interpretability as to how a specific relationship between dependent and independent variables may look. Or it may be that a certain model is assumed or of interest and there is a number of candidate variables that may non-linearly give rise to different model parameter values. We present an approach that combines generalized linear models (GLM) with recursive partitioning that offers enhanced interpretability of classical trees as well as providing an explorative way to assess a candidate variable's influence on a parametric model. This method conducts recursive partitioning of a GLM by (1) fitting the model to the data set, (2) testing for parameter instability over a set of partitioning variables, (3) splitting the data set with respect to the variable associated with the highest instability. The outcome is a tree where each terminal node is associated with a GLM. We will show the method's versatility and suitability to gain additional insight into the relationship of dependent and independent variables by two examples, modelling voting behaviour and a failure model for debt amortization, and compare it to alternative approaches.

[1]  K. Hornik,et al.  Unbiased Recursive Partitioning: A Conditional Inference Framework , 2006 .

[2]  M. Ichinokawa,et al.  Using adaptive area stratification to standardize catch rates with application to North Pacific swordfish (Xiphias gladius) , 2010 .

[3]  João Gama,et al.  Functional Trees , 2001, Machine Learning.

[4]  Heping Zhang,et al.  Recursive Partitioning and Applications , 1999 .

[5]  W. Loh,et al.  Generalized regression trees , 1995 .

[6]  K. Hornik,et al.  Generalized M‐fluctuation tests for parameter instability , 2007 .

[7]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[8]  Kuldeep Kumar,et al.  Robust Statistics, 2nd edn , 2011 .

[9]  L. L. Cam,et al.  Maximum likelihood : an introduction , 1990 .

[10]  H. White.  Maximum Likelihood Estimation of Misspecified Models , 1982 .

[11]  John Hinde,et al.  Statistical Modelling in R , 2009 .

[12]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[13]  K. Hornik,et al.  Model-Based Recursive Partitioning , 2008 .

[14]  H. Ahn.  Tree-structured extreme value model regression , 1994 .

[15]  Eibe Frank,et al.  Logistic Model Trees , 2003, Machine Learning.

[16]  W. Loh,et al.  Tree-structured proportional hazards regression modeling. , 1994, Biometrics.

[17]  W. Loh,et al.  REGRESSION TREES WITH UNBIASED VARIABLE SELECTION AND INTERACTION DETECTION , 2002 .

[18]  Log-gamma regression modeling 'through regression trees , 1996 .

[19]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[20]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[21]  Hongshik Ahn.  Log-normal regression modeling through recursive partitioning , 1996 .

[22]  Xiaogang Su,et al.  Joint Statistical Meetings- Statistical Computing Section Maximum Likelihood Regression Trees , 2022 .

[23]  D. Andrews.  Tests for Parameter Instability and Structural Change with Unknown Change Point , 1993 .

[24]  W. Loh,et al.  LOTUS: An Algorithm for Building Accurate and Comprehensible Logistic Regression Trees , 2004 .

[25]  W. Loh,et al.  Nonparametric estimation of conditional quantiles using quantile regression trees ∗ ( Published in Bernoulli ( 2002 ) , 8 , 561 – 576 ) , 2008 .

[26]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[27]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[28]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[29]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[30]  Ian H. Witten,et al.  Induction of model trees for predicting continuous classes , 1996 .

[31]  J. Morgan,et al.  Problems in the Analysis of Survey Data, and a Proposal , 1963 .

[32]  BUSDM – an algorithm for the bottom-up search of departures from a model , 2011 .

[33]  A. Albert,et al.  On the existence of maximum likelihood estimates in logistic regression models , 1984 .

[34]  Wei-Yin Loh,et al.  Regression by Parts: Fitting Visually Interpretable Models with GUIDE , 2008 .

[35]  H. Keselman,et al.  Multiple Comparison Procedures , 2005 .

[36]  Bertrand Clarke,et al.  Principles and Theory for Data Mining and Machine Learning , 2009 .

[37]  N. Hjort,et al.  Tests For Constancy Of Model Parameters Over Time , 2002 .

[38]  W. Loh,et al.  Improving the precision of classification trees , 2010, 1011.0608.

[39]  P. Chaudhuri,et al.  Piecewise polynomial regression trees , 1994 .

[40]  James J. Chen,et al.  Regression trees for analysis of count data with extra Poisson variation , 2005, Comput. Stat. Data Anal..

[41]  Christina Gloeckner,et al.  Modern Applied Statistics With S , 2003 .

[42]  H. Ahn,et al.  Tree-structured logistic models for over-dispersed binomial data with application to modeling developmental effects. , 1997, Biometrics.

[43]  H. Ahn.  Tree-Structured Exponential Regression Modeling† , 1994 .

[44]  Brian D. Ripley,et al.  Modern applied statistics with S, 4th Edition , 2002, Statistics and computing.

[45]  A. Zeileis.  A Unified Approach to Structural Change Tests Based on ML Scores, F Statistics, and OLS Residuals , 2005 .

[46]  Gabriel Chandler,et al.  Automatic locally adaptive smoothing for tree-based set estimation , 2013 .

[47]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[48]  Kurt Hornik,et al.  Open-source machine learning: R meets Weka , 2009, Comput. Stat..