On Johnson's (2000) Relative Weights Method for Assessing Variable Importance: A Reanalysis

This article provides a reanalysis of J. W. Johnson's (2000) “relative weights” method for assessing variable importance in multiple regression. The primary conclusion of the reanalysis is that the derivation of the method is theoretically flawed and has no more validity than the discredited method of Green, Carroll, and DeSarbo (1978) on which it is based. By means of 2 examples, supplemented by other results from the literature, it is also shown that the method can result in materially distorted inferences when it is compared with another widely used importance metric, namely, general dominance (Azen & Budescu, 2003; Budescu, 1993). Our primary recommendation is that J. W. Johnson's (2000) relative weights method should no longer be used as a variable importance metric for multiple linear regression. In the final section of the article, 2 additional recommendations are made based on our analysis, examples, and discussion.

[1]  D. R. Thomas,et al.  On Measuring the Relative Importance of Explanatory Variables in a Logistic Regression , 2008 .

[2]  J. Bring.  A Geometric Approach to Compare Variables in a Regression Model , 1996 .

[3]  Razia Azen,et al.  Using Dominance Analysis to Determine Predictor Importance in Logistic Regression , 2009 .

[4]  John D. Leeth,et al.  Data Mining CEO Compensation , 2008 .

[5]  P. Sen,et al.  Introduction to bivariate and multivariate analysis , 1981 .

[6]  U. Grömping.  Estimators of Relative Importance in Linear Regression Based on Variance Decomposition , 2007 .

[7]  Razia Azen,et al.  Determining Predictor Importance in Hierarchical Linear Models Using Dominance Analysis , 2013 .

[8]  David V. Budescu,et al.  Beyond Global Measures of Relative Importance: Some Insights from Dominance Analysis , 2004 .

[9]  P. Hoffman.  The paramorphic representation of clinical judgment. , 1960, Psychological bulletin.

[10]  R. M. Johnson.  The minimal transformation to orthonormality , 1966 .

[11]  David V. Budescu,et al.  An Extension of Dominance Analysis to Canonical Correlation Analysis , 2009 .

[12]  J. W. Johnson.  A Heuristic Method for Estimating the Relative Weight of Predictor Variables in Multiple Regression , 2000, Multivariate behavioral research.

[13]  Michael T Braun,et al.  Exploratory regression analysis: A tool for selecting models and determining predictor importance , 2011, Behavior research methods.

[14]  Scott Tonidandel,et al.  Multivariate relative importance: extending relative weight analysis to multivariate criterion spaces. , 2008, The Journal of applied psychology.

[15]  R. D. de Vries,et al.  Leadership = Communication? The Relations of Leaders’ Communication Styles with Leadership Styles, Knowledge Sharing and Leadership Outcomes , 2009, Journal of business and psychology.

[16]  Bruno D. Zumbo,et al.  Using a Measure of Variable Importance to Investigate the Standardization of Discriminant Coefficients , 1996 .

[17]  James M. LeBreton,et al.  Importance of Personality and Job-Specific Affect for Predicting Job Attitudes and Withdrawal Behavior , 2004 .

[18]  U. Grömping.  Dependence of Variable Importance in Random Forests on the Shape of the Regressor Space , 2009 .

[19]  D. R. Thomas.  Interpreting Discriminant Functions: A Data Analytic Approach. , 1992, Multivariate behavioral research.

[20]  Robert E. Ployhart,et al.  A multidimensional approach for evaluating variables in organizational research and practice , 2007 .

[21]  D. Budescu,et al.  The dominance analysis approach for comparing predictors in multiple regression. , 2003, Psychological methods.

[22]  Scott Tonidandel,et al.  Determining the Relative Importance of Predictors in Logistic Regression: An Extension of Relative Weight Analysis , 2010 .

[23]  K. Holzinger,et al.  A study in factor analysis : the stability of a bi-factor solution , 1939 .

[24]  B. B. Jackson.  Comment on “A New Measure of Predictor Variable Importance in Multiple Regression” , 1980 .

[25]  W. Gibson.  Orthogonal Predictors: A Possible Resolution of the Hoffman-Ward Controversy , 1962 .

[26]  B. Zumbo,et al.  Relative Importance of Predictors in Multilevel Modeling , 2014 .

[27]  S. Marshall,et al.  A method to aid in the interpretation of EFA results: An application of Pratt’s measures , 2014 .

[28]  J. Carroll,et al.  A New Measure of Predictor Variable Importance in Multiple Regression , 1978 .

[29]  James M. LeBreton,et al.  Relative Importance Analysis: A Useful Supplement to Regression Analysis , 2011 .

[30]  D. R. Thomas,et al.  On Variable Importance in Linear Regression , 1998 .

[31]  Bruno D. Zumbo,et al.  3 Validity: Foundational Issues and Statistical Methodology , 2006 .

[32]  James M. LeBreton,et al.  History and Use of Relative Importance Indices in Organizational Research , 2004 .

[33]  David V. Budescu,et al.  Comparing Predictors in Multivariate Regression Models: An Extension of Dominance Analysis , 2006 .

[34]  D. Budescu.  Dominance analysis: A new approach to the problem of relative importance of predictors in multiple regression. , 1993 .

[35]  Robert E. Ployhart,et al.  A Monte Carlo Comparison of Relative Importance Methodologies , 2004 .