The Pooled Data Approach for the Estimation of Split-Ballot Multitrait–Multimethod Experiments

Saris, Satorra and Coenders (2004) proposed a new approach for estimating the quality of survey questions, which combines the advantages of the following two existing approaches: the multi-trait-multi-method (MTMM) approach and the split-ballot (SB) approach. In practice, this new approach led to frequent occurrences of non-convergence and improper solutions in the case of a two-group design (Revilla & Saris, 2013). This was very problematic because this approach had been used in many experiments to determine the reliability, validity and method effects of questions that are used in the European Social Survey, which is used in approximately 30 countries. In this article, we propose a new estimation procedure, which we call estimation using pooled data (EUPD), to overcome the problems with this experimental design. We show that EUPD can also be applied in a general setting of structural equation modeling when the sample size is too small for precise estimation but several similar data sets are available.

[1]  M. Browne,et al.  The decomposition of multitrait-multimethod matrices. , 1984, The British journal of mathematical and statistical psychology.

[2]  M. Kuefer.  Psychophysics Introduction To Its Perceptual Neural And Social Prospects , 2016 .

[3]  Michael Eid,et al.  A multitrait-multimethod model with minimal assumptions , 2000 .

[4]  K. Jöreskog.  Statistical analysis of sets of congeneric tests , 1971 .

[5]  A. Satorra.  Asymptotic robust inferences in multi-sample analysis of augmented-moment structures , 1993 .

[6]  Willem E. Saris,et al.  Choosing the Number of Categories in Agree–Disagree Scales , 2014 .

[7]  Duane F. Alwin.  Margins of Error: A Study of Reliability in Survey Measurement , 2007 .

[8]  Robert L. Linn,et al.  Path analysis: Psychological examples. , 1970 .

[9]  Ingwer Borg,et al.  The measurement equivalence of Big Five factor markers for persons with different levels of education. , 2010, Journal of research in personality.

[10]  Willem E. Saris,et al.  Memory effects in MTMM studies , 1995 .

[11]  Lars Kaczmirek,et al.  Assessing Cross-National Equivalence of Measures of Xenophobia: Evidence from Probing in Web Surveys , 2013 .

[12]  Willem E. Saris,et al.  Testing Structural Equation Models or Detection of Misspecifications? , 2009 .

[13]  G. A. Marcoulides,et al.  Advanced structural equation modeling : issues and techniques , 1996 .

[14]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[15]  Kevin J Grimm,et al.  Data Integration Approaches to Longitudinal Growth Modeling , 2017, Educational and psychological measurement.

[16]  F. M. Andrews.  Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach , 1984 .

[17]  Willem E. Saris,et al.  Evaluation of Measurement Instruments Using a Structural Modeling Approach , 2011 .

[18]  Willem E. Saris,et al.  Fit of Different Models for Multitrait-Multimethod Experiments , 2002 .

[19]  Eldad Davidov,et al.  Testing for comparability of human values across countries and time with the third round of the European Social Survey , 2010 .

[20]  A. Satorra,et al.  Power of the likelihood ratio test in covariance structure analysis , 1985 .

[21]  Willem E. Saris,et al.  A comparison of several approximations to the power function of the likelihood ratio test in covariance structure analysis , 1991 .

[22]  Willem E. Saris,et al.  Different Explanations for Correlated Disturbance Terms in MTMM Studies , 2003 .

[23]  A. Satorra.  Fusion of Data Sets in Multivariate Linear Regression with Errors-in-Variables , 1996 .

[24]  Willem E. Saris,et al.  Variation in response functions : a source of measurement error in attitude research , 1988 .

[25]  P. Allison.  Estimation of Linear Models with Incomplete Data , 1987 .

[26]  K. Jöreskog.  A general method for analysis of covariance structures , 1970 .

[27]  Tim Reeskens,et al.  Cross-cultural measurement equivalence of generalized trust. Evidence from the European Social Survey (2002 and 2004) , 2007 .

[28]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research , 2007 .

[29]  A. C. Scherpenzeel.  A Question of Quality. Evaluating survey questions by multitrait-multimethod studies , 1995 .

[30]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research: Saris/Design , 2007 .

[31]  Willem E. Saris,et al.  8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design , 2004 .

[32]  Daniel L. Oberski,et al.  The Quality of the Measurement of Interest in the Political Issues presented in the Media in the ESS , 2008 .

[33]  H. Marsh,et al.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: A Comparison of Alternative Models , 1991 .

[34]  John W Graham,et al.  Planned missing data designs in psychological research. , 2006, Psychological methods.

[35]  R. Cudeck.  Msultiplicative Models and MTMM Matrices , 1988 .

[36]  W. Saris,et al.  The Split-Ballot Multitrait-Multimethod Approach: Implementation and Problems , 2013 .

[37]  Werner Wothke,et al.  Models for Multitratit-Multimethod Matrix Analysis , 1996 .

[38]  R. P. McDonald,et al.  Evaluation of Measurement Instruments by Meta-Analysis of Multitrait Multimethod Studies. , 1992 .

[39]  D. Alwin,et al.  Feeling Thermometers Versus 7-Point Scales , 1997 .