Construct Meaning in Multilevel Settings

We present types of constructs, individual- and cluster-level, and their confirmatory factor analytic validation models when data are from individuals nested within clusters. When a construct is theoretically individual level, spurious construct-irrelevant dependency in the data may appear to signal cluster-level dependency; in such cases, however, and consistent with theory, a single-level analysis with a correction for dependency may be appropriate. Regarding cluster-level constructs, we discuss two types—shared and configural—and present appropriate validation models. Illustrative validation analyses with individual, shared, and configural constructs are provided using empirical data as well as simple simulations demonstrating the spurious effects that can occur with nested data. The article concludes with future directions to be examined in construct validation in multilevel settings.

[1]  A. Meade,et al.  Using Indices of Group Agreement in Multilevel Construct Validation , 2007 .

[2]  Michael J. Zyphur,et al.  Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions , 2008 .

[3]  Kristopher J Preacher,et al.  Reliability estimation in a multilevel confirmatory factor analysis framework. , 2014, Psychological methods.

[4]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[5]  B. Muthén,et al.  Resampling Methods in Mplus for Complex Survey Data , 2010 .

[6]  Laura M. Stapleton,et al.  Design Effects of Multilevel Estimates From National Probability Samples , 2018 .

[7]  S. Kozlowski,et al.  Multilevel Theory, Research, a n d M e t h o d s i n Organizations Foundations, Extensions, and New Directions , 2022 .

[8]  G. Hancock.  Effect size, power, and sample size determination for structured means modeling and mimic approaches to between-groups hypothesis testing of means on a single latent construct , 2001 .

[9]  George A. Marcoulides,et al.  On Multilevel Model Reliability Estimation From the Perspective of Structural Equation Modeling , 2006 .

[10]  Roderick P. McDonald,et al.  Generalizability in Factorable Domains: "Domain Validity and Generalizability"1 , 1978 .

[11]  David B. Dunson,et al.  Bayesian Structural Equation Modeling , 2007 .

[12]  Michael C Neale,et al.  People are variables too: multilevel structural equations modeling. , 2005, Psychological methods.

[13]  Ehri Ryu,et al.  Level-Specific Evaluation of Model Fit in Multilevel Structural Equation Modeling , 2009 .

[14]  Roderick P. McDonald,et al.  THE THEORETICAL FOUNDATIONS OF PRINCIPAL FACTOR ANALYSIS, CANONICAL FACTOR ANALYSIS, AND ALPHA FACTOR ANALYSIS , 1970 .

[15]  Bengt Muthén,et al.  Multilevel Factor Analysis of Class and Student Achievement Components , 1991 .

[16]  R. Brennan,et al.  The Generalizability of Class Means , 1977 .

[17]  Laura M. Stapleton,et al.  The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration , 2014, Educational Psychology Review.

[18]  Tihomir Asparouhov,et al.  Multivariate Statistical Modeling with Survey Data , 2005 .

[19]  Alexander W. Astin,et al.  New measures of college selectivity , 1977 .

[20]  G. A. Marcoulides.  Multilevel Analysis Techniques and Applications , 2002 .

[21]  N. Scrimshaw,et al.  The acceptability of milk and milk products in populations with a high prevalence of lactose intolerance. , 1988, The American journal of clinical nutrition.

[22]  F. Casellas,et al.  Development, Validation, and Applicability of a Symptoms Questionnaire for Lactose Malabsorption Screening , 2009, Digestive Diseases and Sciences.

[23]  Spiridon Penev,et al.  Estimation of maximal reliability for multiple-component instruments in multilevel designs. , 2009, The British journal of mathematical and statistical psychology.

[24]  T. Raykov.  Estimation of Composite Reliability for Congeneric Measures , 1997 .

[25]  D. Chan.  Functional Relations among Constructs in the Same Content Domain at Different Levels of Analysis: A Typology of Composition Models , 1998 .

[26]  Laura M. Stapleton,et al.  An Assessment of Practical Solutions for Structural Equation Modeling with Complex Sample Data , 2006 .

[27]  Wendy M. Yen,et al.  Scaling Performance Assessments: Strategies for Managing Local Item Dependence , 1993 .

[28]  Robert M. O'Brien,et al.  Estimating the Reliability of Aggregate-Level Variables Based on Individual-Level Characteristics , 1990 .

[29]  Lee J. Cronbach,et al.  Research on Classrooms and Schools: Formulation of Questions, Design and Analysis. , 1976 .

[30]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[31]  R. P. McDonald,et al.  Test Theory: A Unified Treatment , 1999 .

[32]  Robert L. Brennan,et al.  The Conventional Wisdom About Group Mean Scores , 1995 .

[33]  D. A. Kenny,et al.  Trait–state models for longitudinal data. , 2001 .

[34]  J. Hox,et al.  Multilevel Structural Equation Models: the Limited Information Approach and the Multivariate Multilevel Approach , 2004 .

[35]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[36]  P. Bliese.  Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. , 2000 .

[37]  David A. Hofmann,et al.  Multilevel Analytical Techniques: Commonalties, Differences, and Continuing Questions. , 2000 .

[38]  Ulrich Trautwein,et al.  Classroom Climate and Contextual Effects: Conceptual and Methodological Issues in the Evaluation of Group-Level Effects , 2012 .

[39]  Ulrich Trautwein,et al.  A 2 × 2 taxonomy of multilevel latent contextual models: accuracy-bias trade-offs in full and partial error correction models. , 2011, Psychological methods.

[40]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[41]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[42]  J. Schweig.  Multilevel Factor Analysis by Model Segregation , 2014 .

[43]  T. Asparouhov.  General Random Effect Latent Variable Modeling : Random Subjects , Items , Contexts , and Parameters , 2012 .

[44]  Thanh Le,et al.  Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K): Combined User's Manual for the ECLS-K Eighth-Grade and K-8 Full Sample Data Files and Electronic Codebooks. NCES 2009-004. , 2009 .

[45]  Laura M. Stapleton.  Variance Estimation Using Replication Methods in Structural Equation Modeling With Complex Sample Data , 2008 .

[46]  Spiridon Penev,et al.  Evaluation of Reliability Coefficients for Two-Level Models via Latent Variable Analysis , 2010 .

[47]  Jeroen K. Vermunt,et al.  Composing Group-Level Constructs From Individual-Level Survey Data , 2009 .

[48]  Tihomir Asparouhov,et al.  Multilevel Modeling of Complex Survey Data , 2006 .

[49]  Daniel J. Pratt,et al.  Education Longitudinal Study of 2002: Base Year Data File User's Manual. NCES 2004-405. , 2004 .

[50]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[51]  T. Brown.  Confirmatory factor analysis for applied research, 2nd ed. , 2015 .

[52]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[53]  Kanreutai Klangphahol.  Applying multilevel confirmatory factor analysis techniques to perceived homework quality , 2009 .

[54]  Laura M. Stapleton,et al.  The Reviewer’s Guide to Quantitative Methods in the Social Sciences , 2018 .

[55]  P. Greenbaum,et al.  Multilevel Confirmatory Factor Analysis of a Scale Measuring Interagency Collaboration of Children’s Mental Health Agencies , 2011, Journal of emotional and behavioral disorders.

[56]  B. Zumbo,et al.  Validation of Multilevel Constructs: Validation Methods and Empirical Findings for the EDI , 2011 .

[57]  Terry E. Duncan,et al.  Analyzing measurement models of latent variables through multilevel confirmatory factor analysis and hierarchical linear modeling approaches , 1998 .