Modelling method effects as individual causal effects

Method effects often occur when different methods are used for measuring the same construct. We present a new approach for modelling this kind of phenomenon, consisting of a definition of method effects and a first model, the method effect model, that can be used for data analysis. This model may be applied to multitrait-multimethod data or to longitudinal data where the same construct is measured with at least two methods at all occasions. In this new approach, the definition of the method effects is based on the theory of individual causal effects by Neyman and Rubin. Method effects are accordingly conceptualized as the individual effects of applying measurement method j instead of k. They are modelled as latent difference scores in structural equation models. A reference method needs to be chosen against which all other methods are compared. The model fit is invariant to the choice of the reference method. The model allows the estimation of the average of the individual method effects, their variance, their correlation with the traits (and other latent variables) and the correlation of different method effects among each other. Furthermore, since the definition of the method effects is in line with the theory of causality, the method effects may (under certain conditions) be interpreted as causal effects of the method. The method effect model is compared with traditional multitrait-multimethod models. An example illustrates the application of the model to longitudinal data analysing the effect of negatively (such as 'feel bad') as compared with positively formulated items (such as 'feel good') measuring mood states.

[1]  M. Eid,et al.  Modeling true intraindividual change: True change as a latent variable. , 1997 .

[2]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[3]  H. Hops,et al.  Multimethod Comparison of Similarity in School Adjustment of Siblings and Unrelated Children , 1993 .

[4]  Michael Eid,et al.  Latent state–trait theory and research in personality and individual differences , 1999 .

[5]  B. Byrne,et al.  Do We See Ourselves as Others Infer: A Comparison of Self-Other Agreement on Multiple Dimensions of Self-Concept from Two Continents. , 1993 .

[6]  J. Russell,et al.  Independence and bipolarity in the structure of current affect. , 1998 .

[7]  M. Eid,et al.  Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model. , 2003, Psychological methods.

[8]  J. M. Carroll,et al.  On the bipolarity of positive and negative affect. , 1999, Psychological bulletin.

[9]  Raymond L. Debus,et al.  Effects of internally focused feedback and attributional feedback on enhancement of academic self-concept , 1991 .

[10]  J. A. Gómez-Fraguela,et al.  Assessment of the Validity of Parenting Constructs Using the Multitrait-Multimethod Model , 2006 .

[11]  K. Widaman.  Hierarchically Nested Covariance Structure Models for Multitrait-Multimethod Data , 1985 .

[12]  Herbert W. Marsh,et al.  Latent variable models of multitrait-multimethod data. , 1995 .

[13]  R. Steyer,et al.  Is it Possible to Feel Good and Bad at the Same Time? New Evidence on the Bipolarity of Mood-state Dimensions , 2004 .

[14]  E. Raufaste,et al.  Imperfect or Perfect Dynamic Bipolarity? The Case of Antonymous Affective Judgments , 2005 .

[15]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[16]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[17]  Robert L. Holzbach,et al.  Rater bias in performance ratings: Superior, self-, and peer ratings. , 1978 .

[18]  T. Ortner.  On changing the position of items in personality questionnaires: Analysing effects of item sequence using IRT. , 2004 .

[19]  Rolf Steyer,et al.  Models of classical psychometric test theory as stochastic measurement models: Representation, uniqueness, meaningfulness, identifiability, and testability. , 1989 .

[20]  D. Cole,et al.  Relation of depressive symptoms to the structure of self-knowledge in childhood. , 1996, Journal of abnormal psychology.

[21]  Karl G. Jöreskog,et al.  Analyzing psychological data by structural analysis of covariance matrices , 1974 .

[22]  L. F. Barrett,et al.  Intimacy as an interpersonal process: the importance of self-disclosure, partner disclosure, and perceived partner responsiveness in interpersonal exchanges. , 1998, Journal of personality and social psychology.

[23]  R. Motl,et al.  Longitudinal Invariance of Self-Esteem and Method Effects Associated With Negatively Worded Items , 2002 .

[24]  M. R. Novick.  The axioms and principal results of classical test theory , 1965 .

[25]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[26]  Donald W. Zimmerman.  Probability spaces, hilbert spaces, and the axioms of test theory , 1975 .

[27]  Allen I. Kraut,et al.  Some effects of position on opinion survey items. , 1975 .

[28]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[29]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[30]  M. Mount,et al.  Psychometric properties of subordinate ratings of managerial performance. , 1984 .

[31]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[32]  Markus Heinrichs,et al.  Der Mehrdimensionale Befindlichkeitsfragebogen (MDBF) , 2002 .

[33]  Manfred J. Schmitt.  Conceptual, Theoretical, and Historical Foundations of Multimethod Assessment. , 2006 .

[34]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[35]  H W Marsh,et al.  Positive and negative global self-esteem: a substantively meaningful distinction or artifactors? , 1996, Journal of personality and social psychology.

[36]  R. Devellis.  Classical Test Theory , 2006, Medical care.

[37]  A. McConnell,et al.  Relations among the Implicit Association Test, Discriminatory Behavior, and Explicit Measures of Racial Attitudes , 2001 .

[38]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[39]  Steven E. Scullen,et al.  Bias in the Correlated Uniqueness Model for MTMM Data , 2004 .

[40]  D. A. Kenny,et al.  An empirical application of confirmatory factor analysis to the multitrait-multimethod matrix , 1976 .

[41]  D. Cole,et al.  Modeling causal relations between academic and social competence and depression: a multitrait-multimethod longitudinal study of children. , 1996, Journal of abnormal psychology.

[42]  Gilles E. Gignac.  Evaluating subtest ‘g’ saturation levels via the single trait-correlated uniqueness (STCU) SEM approach: Evidence in favor of crystallized subtests as the best indicators of ‘g’ , 2006 .

[43]  Michael Eid,et al.  A multitrait-multimethod model with minimal assumptions , 2000 .

[44]  K. Jöreskog.  Simultaneous factor analysis in several populations , 1971 .

[45]  R. Steyer.  Analyzing Individual and Average Causal Effects via Structural Equation Models , 2005 .

[46]  H. Marsh,et al.  Self-Other Agreement on Multiple Dimensions of Preadolescent Self-Concept: Inferences by Teachers, Mothers, and Fathers. , 1991 .

[47]  Christine DiStefano,et al.  Wording Effects in Self-Esteem Scales: Methodological Artifact or Response Style? , 2003 .

[48]  R. Steyer,et al.  The true-change model with individual method effects: reliability issues. , 2008, The British journal of mathematical and statistical psychology.

[49]  R. Steyer,et al.  States and traits in psychological assessment. , 1992 .