The Role of Model Complexity in the Evaluation of Structural Equation Models

This dissertation represents an investigation into the role of model complexity in structural equation modeling (SEM) and how traditional notions of model fit, which do not typically consider complexity, are inadequate for summarizing the success or failure of a model. Model fit is traditionally summarized in an index which communicates the agreement between a given model and a particular data set. However, demonstrating that a given model could have generated a given data set is not sufficient to show that it is a good model. Because complexity partially determines the ability of a model to fit data, model complexity should not be ignored when evaluating model fit. The importance of model complexity is examined in the SEM context by simulating correlation matrices and applying a series of models which differ in complexity. First, it is shown that models differing in the number of free parameters can fit the same random data differentially well by a simple criterion of good fit. Second, models with the same number of free parameters, but which differ in functional form, are found to fit the same data differentially well. Third, it is found that restrictions placed on parameter range have an impact on model complexity. Because traditional fit indices usually correct for model complexity due only to the number of free parameters, these findings have important implications for how models are assessed and compared in the SEM paradigm.

[1]  J. Dunn Model complexity: The fit to random data reconsidered , 2000, Psychological research.

[2]  P. Costa,et al.  Revised NEO Personality Inventory (NEO-PI-R) and NEO-Five-Factor Inventory (NEO-FFI) , 1992 .

[3]  Michael McAleer,et al.  Simplicity, scientific inference and econometric modelling , 1995 .

[4]  J. Cutting,et al.  Selectivity, scope, and simplicity of models: a lesson from fitting judgments of perceived depth. , 1992, Journal of experimental psychology. General.

[5]  N. Campbell,et al.  Scientific Inference , 1931, Nature.

[6]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[7]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[8]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[9]  Hongyuan Zha,et al.  A note on constructing a symmetric matrix with specified diagonal entries and eigenvalues , 1995 .

[10]  N. N. Chan,et al.  Diagonal elements and eigenvalues of a real symmetric matrix , 1983 .

[11]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[12]  R. Cudeck Msultiplicative Models and MTMM Matrices , 1988 .

[13]  R. Holmes On random correlation matrices , 1991 .

[14]  S. Mulaik,et al.  EVALUATION OF GOODNESS-OF-FIT INDICES FOR STRUCTURAL EQUATION MODELS , 1989 .

[15]  Phillip L. Ackerman,et al.  Current-events knowledge in adults: an investigation of age, intelligence, and nonability determinants. , 2001 .

[16]  J D Botha,et al.  Uniform Indices-of-Fit for Factor Analysis Models. , 1988, Multivariate behavioral research.

[17]  I. J. Myung,et al.  Counting probability distributions: Differential geometry and model selection , 2000, Proc. Natl. Acad. Sci. USA.

[18]  I. J. Myung,et al.  The Importance of Complexity in Model Selection. , 2000, Journal of mathematical psychology.

[19]  Herbert W. Marsh,et al.  Goodness of fit in confirmatory factor analysis: The effects of sample size and model parsimony , 1994 .

[20]  L. Tucker,et al.  Evaluation of factor analytic research procedures by means of simulated correlation matrices , 1969 .

[21]  Werner Wothke,et al.  Models for Multitratit-Multimethod Matrix Analysis , 1996 .

[22]  M. McAleer,et al.  The complexity of simplicity , 1997 .

[23]  P. Odell,et al.  A Numerical Procedure to Generate a Sample Covariance Matrix , 1966 .

[24]  R. MacCallum,et al.  The use of causal indicators in covariance structure models: some practical issues. , 1993, Psychological bulletin.

[25]  I. J. Myung,et al.  When a good fit can be bad , 2002, Trends in Cognitive Sciences.

[26]  K. Bollen A New Incremental Fit Index for General Structural Equation Models , 1989 .

[27]  H. O. Hartley,et al.  Monte Carlo Computations in Normal Correlation Problems , 1963, JACM.

[28]  P. Ackerman,et al.  Personality-intelligence relations: assessment of typical intellectual engagement , 1992 .

[29]  Francis Sullivan,et al.  The Metropolis Algorithm , 2000, Computing in Science & Engineering.

[30]  Grünwald,et al.  Model Selection Based on Minimum Description Length. , 2000, Journal of mathematical psychology.

[31]  M. Browne,et al.  The decomposition of multitrait-multimethod matrices. , 1984, The British journal of mathematical and statistical psychology.

[32]  J. Rissanen Stochastic complexity and the mdl principle , 1987 .

[33]  K. Jöreskog A general approach to confirmatory maximum likelihood factor analysis , 1969 .

[34]  Arnold Zellner,et al.  The enigma of simplicity , 2001 .

[35]  Woojae Kim,et al.  Flexibility versus generalizability in model selection , 2003, Psychonomic bulletin & review.

[36]  H Pashler,et al.  How persuasive is a good fit? A comment on theory testing. , 2000, Psychological review.

[37]  D. Rindskopf Using phantom and imaginary latent variables to parameterize constraints in linear structural models , 1984 .

[38]  J. H. Steiger Statistically based tests for the number of common factors , 1980 .

[39]  F. Hohn,et al.  Elementary Matrix Algebra , 1959 .

[40]  Joseph Lee Rodgers,et al.  Theory development should begin (but not end) with good empirical fits: a comment on Roberts and Pashler (2000). , 2002, Psychological review.

[41]  Cutting,et al.  Accuracy, Scope, and Flexibility of Models. , 2000, Journal of mathematical psychology.

[42]  Shaobo Zhang Applications of geometric complexity and the minimum description length principle in mathematical modeling of cognition , 2000 .

[43]  Golden,et al.  Statistical Tests for Comparing Possibly Misspecified and Nonnested Models. , 2000, Journal of mathematical psychology.

[44]  Browne,et al.  Cross-Validation Methods. , 2000, Journal of mathematical psychology.

[45]  Craig B. Borkowf,et al.  Random Number Generation and Monte Carlo Methods , 2000, Technometrics.

[46]  Cullen Schaffer Overfitting avoidance as bias , 2004, Machine Learning.

[47]  S. Bem The measurement of psychological androgyny. , 1974, Journal of consulting and clinical psychology.

[48]  Philip I. Davies,et al.  Numerically Stable Generation of Correlation Matrices and Their Factors , 2000 .

[49]  B. Byrne A Primer of Lisrel: Basic Applications and Programming for Confirmatory Factor Analytic Models , 1989 .

[50]  James C. Anderson,et al.  Monte Carlo Evaluations of Goodness of Fit Indices for Structural Equation Models , 1992 .

[51]  2 Models for Multitrait-Multimethod Matrices , 1996 .

[52]  C. Schriesheim Causal Analysis: Assumptions, Models, and Data , 1982 .

[53]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.

[54]  Arnold Zellner,et al.  Simplicity, Inference and Modelling , 2002 .

[55]  J. Stanley,et al.  Restrictions on the Possible Values of r12, Given r13 and r23 , 1969 .

[56]  Leon Jay Gleser A Canonical Representation for the Noncentral Wishart Distribution Useful for Simulation , 1976 .

[57]  Bamber,et al.  How to Assess a Model's Testability and Identifiability. , 2000, Journal of mathematical psychology.

[58]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[59]  A. J. Swain,et al.  Analysis of parametric structures for variance matrices / by Anthony J. Swain , 1975 .

[60]  David R. Anderson,et al.  Model Selection and Multimodel Inference , 2003 .

[61]  P. Bentler,et al.  Fit indices in covariance structure modeling : Sensitivity to underparameterized model misspecification , 1998 .

[62]  P. Bentler,et al.  Significance Tests and Goodness of Fit in the Analysis of Covariance Structures , 1980 .

[63]  L. Tucker,et al.  A reliability coefficient for maximum likelihood factor analysis , 1973 .

[64]  Shang P. Lin,et al.  Generation of Population Correlation Matrices with Specified Eigenvalues , 1985 .

[65]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[66]  D. G. Johnson,et al.  The generation of pseudo- random correlation matrices , 1980 .

[67]  Arnold Zellner,et al.  Simplicity and statistical inference , 2002 .

[68]  R. P. McDonald,et al.  Choosing a multivariate model: Noncentrality and goodness of fit. , 1990 .

[69]  Karl G. Jöreskog,et al.  Lisrel 8: User's Reference Guide , 1997 .

[70]  R. P. McDonald,et al.  An index of goodness-of-fit based on noncentrality , 1989 .

[71]  M. Browne,et al.  The direct product model for the mtmm matrix parameterized as a second order factor analysis model , 1990 .

[72]  Mervyn G. Marasinghe,et al.  Direct methods for generating extreme characteristic roots of certain random matrices , 1982 .

[73]  R. MacCallum,et al.  When fit indices and residuals are incompatible. , 2002, Psychological methods.

[74]  R. MacCallum,et al.  Power Analysis in Covariance Structure Modeling Using GFI and AGFI. , 1997, Multivariate behavioral research.

[75]  H. Bozdogan,et al.  Akaike's Information Criterion and Recent Developments in Information Complexity. , 2000, Journal of mathematical psychology.

[76]  K. Bollen Multiple indicators: Internal consistency or no necessary relationship? , 1984 .

[77]  Robert C MacCallum,et al.  2001 Presidential Address: Working with Imperfect Models , 2003, Multivariate behavioral research.

[78]  Stephan Lewandowsky,et al.  Using Parameter Sensitivity and Interdependence to Predict Model Scope and Falsifiability , 1996 .

[79]  R. Schulz,et al.  Social support as a mediator in the relation between functional status and quality of life in older adults. , 1996, Psychology and aging.

[80]  R. Nickerson Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[81]  I. Olkin,et al.  Generating Correlation Matrices , 1984 .

[82]  M. Forster The new science of simplicity 1 , 2001 .

[83]  Peter M. Bentler,et al.  EQS : structural equations program manual , 1989 .

[84]  Stanley A. Mulaik,et al.  The Curve-Fitting Problem: An Objectivist View , 2001, Philosophy of Science.

[85]  M. Forster,et al.  Key Concepts in Model Selection: Performance and Generalizability. , 2000, Journal of mathematical psychology.

[86]  W. Quine The ways of paradox, and other essays , 1966 .

[87]  C. Chalmers Generation of correlation matrices with a given eigen–structure , 1975 .

[88]  A. Afifi,et al.  Comparison of Stopping Rules in Forward “Stepwise” Regression , 1977 .

[89]  Elliott Sober,et al.  Simplicity, Inference and Modelling: What is the problem of simplicity? , 2002 .

[90]  Peter Green,et al.  Markov chain Monte Carlo in Practice , 1996 .

[91]  R. MacCallum,et al.  Power analysis and determination of sample size for covariance structure modeling. , 1996 .

[92]  D. W. D.Sc.,et al.  XLII. On certain fundamental principles of scientific inquiry , 1921 .

[93]  Charles E. Collyer,et al.  Comparing strong and weak models by fitting them to computer-generated data , 1985 .

[94]  Jorma Rissanen,et al.  Strong optimality of the normalized ML models as universal codes and information in data , 2001, IEEE Trans. Inf. Theory.

[95]  Malcolm R. Forster,et al.  How to Tell When Simpler, More Unified, or Less Ad Hoc Theories will Provide More Accurate Predictions , 1994, The British Journal for the Philosophy of Science.

[96]  Clifford M. Hurvich Mean Square over Degrees of Freedom: New Perspectives on a Model Selection Treasure , 1997 .

[97]  I. J. Myung,et al.  GUEST EDITORS' INTRODUCTION: Special Issue on Model Selection , 2000 .

[98]  M. Levitt,et al.  Close relationships in mothers of distressed and normal newborns : support, expectancy confirmation and maternal well-being , 1991 .

[99]  Moody T. Chu,et al.  Constructing a Hermitian Matrix from Its Diagonal Entries and Eigenvalues , 1995, SIAM J. Matrix Anal. Appl..

[100]  M. R. Mickey,et al.  Population correlation matrices for sampling experiments , 1978 .

[101]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[102]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .