The Issue of Statistical Power for Overall Model Fit in Evaluating Structural Equation Models: Examples from Industrial-Organizational Psychology Research

Statistical power is an important concept for psychological research. However, examining the power of a structural equation model (SEM) is rare in practice. This article provides an accessible review of the concept of statistical power for the Root Mean Square Error of Approximation (RMSEA) index of overall model fit in structural equation modeling. By way of example, we examine the current state of power in the literature by reviewing studies in top Industrial-Organizational (I/O) Psychology journals using SEMs. Results indicate that in many studies, power is very low, which implies acceptance of invalid models. Additionally, we examined methodological situations which may have an influence on statistical power of SEMs. Results showed that power varies significantly as a function of model type and whether or not the model is the main model for the study. Finally, results indicated that power is significantly related to model fit statistics used in evaluating SEMs. The results from this quantitative review imply that researchers should be more vigilant with respect to power in structural equation modeling. We therefore conclude by offering methodological best practices to increase confidence in the interpretation of structural equation modeling results with respect to statistical power issues.

[1]  Kenneth A. Bollen,et al.  Modeling Strategies: In Search of the Holy Grail , 2000 .

[2]  Structural Equation Modeling Power and Sample Size for the Root Mean Square Error of Approximation Test of not Close Fit in , 2001 .

[3]  James C. Anderson,et al.  STRUCTURAL EQUATION MODELING IN PRACTICE: A REVIEW AND RECOMMENDED TWO-STEP APPROACH , 1988 .

[4]  Shaun McQuitty,et al.  Statistical power and structural equation models in business research , 2004 .

[5]  J. Pearl.  Interpretation and Identification of Causal Mediation , 2013, Psychological methods.

[6]  Taehun Lee,et al.  The Issue of Isopower in Power Analysis for Tests of Structural Equation Models , 2010 .

[7]  David A. Specht,et al.  On the evaluation of causal models , 1975 .

[8]  B. Thompson,et al.  EFFECTS OF SAMPLE SIZE, ESTIMATION METHODS, AND MODEL SPECIFICATION ON STRUCTURAL EQUATION MODELING FIT INDEXES , 1999 .

[9]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[10]  R. MacCallum,et al.  Power analysis and determination of sample size for covariance structure modeling. , 1996 .

[11]  B. Muthén,et al.  How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power , 2002 .

[12]  David R. Cox,et al.  Approximations to noncentral distributions , 1987 .

[13]  Larry J. Williams,et al.  The Myth of Global Fit Indices and Alternatives for Assessing Latent Variable Relations , 2011 .

[14]  Elwood F. Holton,et al.  Research in Organizations: Foundations and Methods of Inquiry , 2005 .

[15]  Peter M. Bentler,et al.  Practical Issues in Structural Modeling , 1987 .

[16]  R. MacCallum,et al.  Applications of structural equation modeling in psychological research. , 2000, Annual review of psychology.

[17]  VAN ALSTINE,et al.  Evaluation of Goodness-ofFit Indices for Structural Equation Models , 2001 .

[18]  P. Patnaik.  THE NON-CENTRAL χ2- AND F-DISTRIBUTIONS AND THEIR APPLICATIONS , 1949 .

[19]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[20]  Robert C. MacCallum,et al.  Effect of Estimation Method on Incremental Fit Indexes for Covariance Structure Models , 1993 .

[21]  M. Browne,et al.  Single Sample Cross-Validation Indices for Covariance Structures. , 1989, Multivariate behavioral research.

[22]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[23]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[24]  K. Bollen,et al.  An Empirical Evaluation of the Use of Fixed Cutoff Points in RMSEA Test Statistic in Structural Equation Models , 2008, Sociological methods & research.

[25]  Richard P. Bagozzi,et al.  Specification, evaluation, and interpretation of structural equation models , 2012 .

[26]  R. P. McDonald,et al.  Structural Equations with Latent Variables , 1989 .

[27]  P. Barrett.  Structural equation modelling : Adjudging model fit , 2007 .

[28]  P. Bentler,et al.  Significance Tests and Goodness of Fit in the Analysis of Covariance Structures , 1980 .

[29]  L. Hayduk,et al.  Testing! testing! one, two, three - Testing the theory in structural equation models! , 2007 .

[30]  Niels G Waller,et al.  The path analysis controversy: a new statistical approach to strong appraisal of verisimilitude. , 2002, Psychological methods.

[31]  J. S. Long,et al.  Testing Structural Equation Models , 1993 .

[32]  J. H. Steiger.  Structural Model Evaluation and Modification: An Interval Estimation Approach. , 1990, Multivariate behavioral research.

[33]  R. Bagozzi,et al.  On the evaluation of structural equation models , 1988 .

[34]  J. Pearl.  The Causal Mediation Formula—A Guide to the Assessment of Pathways and Mechanisms , 2012, Prevention Science.

[35]  Kevin Kim,et al.  The Relation Among Fit Indexes, Power, and Sample Size in Structural Equation Modeling , 2005 .

[36]  S. Mulaik,et al.  EVALUATION OF GOODNESS-OF-FIT INDICES FOR STRUCTURAL EQUATION MODELS , 1989 .

[37]  Gregory R. Hancock,et al.  Structural equation modeling : a second course , 2006 .

[38]  J. H. Steiger.  Statistically based tests for the number of common factors , 1980 .

[39]  Leslie A. Hayduk,et al.  LISREL Issues, Debates and Strategies , 1996 .

[40]  R. Landis,et al.  The Earth Is Not Round (p = .00) , 2011 .

[41]  Peter M Bentler,et al.  Quantified choice of root-mean-square errors of approximation for evaluation and power analysis of small differences between structural equation models. , 2011, Psychological methods.

[42]  Fritz Drasgow,et al.  Assessing Goodness of Fit: Simple Rules of Thumb Simply Do Not Work , 2011 .

[43]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[44]  J. Pearl.  Causal inference in statistics: An overview , 2009 .

[45]  Edward E. Rigdon,et al.  Calculating degrees of freedom for a structural equation model , 1994 .

[46]  R. Vandenberg,et al.  On the Practice of Allowing Correlated Residuals Among Indicators in Structural Equation Models , 2010 .

[47]  Larry J. Williams,et al.  Parsimony‐based fit indices for multiple‐indicator models: Do they work? , 1994 .

[48]  Huy Le,et al.  The problem of empirical redundancy of constructs in organizational research: An empirical investigation , 2010 .

[49]  R. MacCallum,et al.  Power Analysis in Covariance Structure Modeling Using GFI and AGFI. , 1997, Multivariate behavioral research.

[50]  Cameron N. McIntosh,et al.  Rethinking fit assessment in structural equation modelling: A commentary and elaboration on Barrett (2007) , 2007 .

[51]  R. MacCallum,et al.  Testing differences between nested covariance structure models: Power analysis and null hypotheses. , 2006, Psychological methods.

[52]  David W. Gerbing,et al.  Assumptions and Comparative Strengths of the Two-Step Approach , 1992 .

[53]  Youjae Yi,et al.  Assumptions of the Two-Step Approach to Latent Variable Modeling , 1992 .

[54]  Ulman Lindenberger,et al.  Statistical Power in Structural Equation Modeling , 2007 .

[55]  H. Marsh,et al.  In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings , 2004 .

[56]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[57]  R. D. Lowrey.  Modeling strategies , 1995 .

[58]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[59]  T.S.J.G. B¿gels,et al.  Publish or Perish , 2005 .