A Monte Carlo Examination of an MTMM Model With Planned Incomplete Data Structures

The classic approach for partitioning and assessing reliability and validity has been through the use of the multitrait-multimethod (MTMM) model. The MTMM approach generally involves 3 different groups (method) evaluating 3 traits. This approach can be reconceptualized for questionnaire evaluation, so that the method becomes 3 different scaling types, which are administered to the same respondents on different occasions to avoid carryover effects. A serious limitation of this MTMM model is that data are required from respondents on at least 3 different occasions, thus placing a heavy burden on the researcher and respondents. Planned incomplete data designs for the purpose of substantially reducing the amount of data required for MTMM models were investigated: 1st, a design that reduces the amount of data collected at the 3rd administration by 22%; and 2nd, a design in which data need only be collected at 2 occasions. The performance of Listwise Deletion, Pairwise Deletion, and the expectation maximization (EM) algorithm at dealing with planned incomplete data are examined through a series of simulations. Results indicate that EM was generally precise and efficient.

[1]  Jürgen Baumert,et al.  Modeling longitudinal and multilevel data: Practical issues, applied approaches, and specific examples. , 2000 .

[2]  A. Satorra,et al.  Scale dependence of the True Score MTMM model , 2004 .

[3]  John W. Graham,et al.  Planned missing-data designs in analysis of change. , 2001 .

[4]  D P MacKinnon,et al.  Maximizing the Usefulness of Data Obtained with Planned Missing Value Patterns: An Application of Maximum Likelihood Procedures. , 1996, Multivariate behavioral research.

[5]  J J McArdle,et al.  Structural Factor Analysis Experiments with Incomplete Data. , 1994, Multivariate behavioral research.

[6]  D. Rubin.  INFERENCE AND MISSING DATA , 1975 .

[7]  Craig K. Enders,et al.  A Primer on Maximum Likelihood Algorithms Available for Use With Missing Data , 2001 .

[8]  Nicole A. Lazar,et al.  Statistical Analysis With Missing Data , 2003, Technometrics.

[9]  Willem E. Saris,et al.  Memory effects in MTMM studies , 1995 .

[10]  R. P. McDonald,et al.  Evaluation of Measurement Instruments by Meta-Analysis of Multitrait Multimethod Studies. , 1992 .

[11]  J L Schafer,et al.  Multiple Imputation for Multivariate Missing-Data Problems: A Data Analyst's Perspective. , 1998, Multivariate behavioral research.

[12]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[13]  J. Baumert,et al.  Longitudinal and multi-group modeling with missing data , 2022 .

[14]  Roger L. Brown.  Efficacy of the indirect approach for estimating structural equation models with missing data: A comparison of five methods , 1994 .

[15]  J. Graham,et al.  Analysis with missing data in drug prevention research. , 1994, NIDA research monograph.

[16]  Marley W. Watkins,et al.  SPSS Software , 2022, A Step-by-Step Guide to Exploratory Factor Analysis with SPSS.

[17]  Bengt Muthén,et al.  On structural equation modeling with data that are not missing completely at random , 1987 .

[18]  M. Raymond.  Missing Data in Evaluation Research , 1986 .

[19]  J. Graham,et al.  Preventing alcohol, marijuana, and cigarette use among adolescents: peer pressure resistance training versus establishing conservative norms. , 1991, Preventive medicine.

[20]  Fred S. Switzer,et al.  A Monte Carlo Analysis of Missing Data Techniques in a HRM Setting , 1995 .