Measurement of composite reliability in research using partial least squares: some issues and an alternative approach

The accurate estimation of reliability is of great importance to the conduct and interpretation of empirical research as it is used to judge the quality of reported research, often plays a role in publication decisions, and is a key element of meta-analytic reviews. When employing partial least squares (PLS) as the method of analysis, the reliability of the composites involved in the model is typically the parameter examined. In this research, we describe the existence of three important issues concerning the accuracy of composite reliability estimation in PLS analysis: the assumption of equal indicator weights, the bias in loading estimates, and the lack of independence between indicator loadings and weights. We subsequently present an alternative approach to correct these issues. Using a Monte Carlo simulation we provide a demonstration of both the effects of these issues on research decisions and the improved accuracy of the alternative method.

[1]  B. Thompson.  Score Reliability: Contemporary Thinking on Reliability Issues , 2002 .

[2]  Heng Li.  A unifying expression for the maximal reliability of a linear composite , 1997 .

[3]  A. Kaplan,et al.  A Beginner's Guide to Partial Least Squares Analysis , 2004 .

[4]  Wynne W. Chin.  The partial least squares approach for structural equation modeling. , 1998 .

[5]  Donald B. Rubin,et al.  Reliability of measurement in psychology: From Spearman-Brown to maximal reliability. , 1996 .

[6]  Detmar W. Straub,et al.  Validating Instruments in MIS Research , 1989, MIS Q..

[7]  Wynne W. Chin,et al.  A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte Carlo Simulation Study and Voice Mail Emotion/Adoption Study , 1996, ICIS.

[8]  R. Lennox,et al.  Conventional wisdom on measurement: A structural equation perspective. , 1991 .

[9]  T. Raykov.  Estimation of Composite Reliability for Congeneric Measures , 1997 .

[10]  T. Raykov.  Analytic Estimation of Standard Error and Confidence Interval for Scale Reliability , 2002, Multivariate behavioral research.

[11]  Donald R. Bacon,et al.  Composite Reliability in Structural Equations Modeling , 1995 .

[12]  Matti Rossi,et al.  Mobile Games: Analyzing the Needs and Values of the Consumers , 2010 .

[13]  Julian C. Stanley,et al.  Differential Weighting: A Review of Methods and Empirical Studies1 , 1970 .

[14]  Tenko Raykov,et al.  A Method for Obtaining Standard Errors and Confidence Intervals of Composite Reliability for Congeneric Items , 1998 .

[15]  Jason Bennett Thatcher,et al.  The Diffusion of Second-Generation Statistical Techniques in Information Systems Research from 1990-2008 , 2010 .

[16]  Detmar W. Straub,et al.  Structural Equation Modeling and Regression: Guidelines for Research Practice , 2000, Commun. Assoc. Inf. Syst..

[17]  R. Hoyle.  Statistical Strategies for Small Sample Research , 1999 .

[18]  Patrick E. Shrout,et al.  Reliability of Scales With General Structure: Point and Interval Estimation Using a Structural Equation Modeling Approach , 2002 .

[19]  P. Gagné,et al.  Measurement Model Quality, Sample Size, and Solution Propriety in Confirmatory Factor Models , 2006, Multivariate behavioral research.

[20]  Tenko Raykov,et al.  Estimation of maximal reliability: a note on a covariance structure modelling approach. , 2004, The British journal of mathematical and statistical psychology.

[21]  Spiridon Penev,et al.  On the Relationship Between Maximal Reliability and Maximal Validity of Linear Composites , 2006, Multivariate behavioral research.

[22]  Wynne W. Chin,et al.  A critical look at partial least squares modeling , 2009 .

[23]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[24]  The theory of test validity and correlated errors of measurement , 1977 .

[25]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[26]  L. S. Feldt.  Estimating the Reliability of a Test Battery Composite or a Test Score Based on Weighted Item Scoring , 2004 .

[27]  Wynne W. Chin,et al.  A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte Carlo Simulation Study and an Electronic - Mail Emotion/Adoption Study , 2003, Inf. Syst. Res..

[28]  R. Sitgreaves.  Psychometric theory (2nd ed.). , 1979 .

[29]  C. I. Mosier,et al.  On the reliability of a weighted composite , 1943 .

[30]  B. Muthén,et al.  How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power , 2002 .

[31]  Kenneth A. Bollen,et al.  Evaluating Effect, Composite, and Causal Indicators in Structural Equation Models , 2011, MIS Q..

[32]  Jukka Ylitalo,et al.  International Conference on Information Systems ( ICIS ) 2010 CONSTRUCT VALIDITY IN PARTIAL LEAST SQUARES PATH MODELING , 2017 .

[33]  D. W. Drewes,et al.  Beyond the Spearman-Brown: a structural approach to maximal reliability. , 2000, Psychological methods.

[34]  Kristopher J Preacher,et al.  Sample Size in Factor Analysis: The Role of Model Error , 2001, Multivariate behavioral research.

[35]  Tenko Raykov,et al.  Bias of Coefficient afor Fixed Congeneric Measures with Correlated Errors , 2001 .

[36]  D. Straub,et al.  Editor's comments: a critical look at the use of PLS-SEM in MIS quarterly , 2012 .

[37]  Susan M. Case,et al.  The Reliability and Validity of Weighted Composite Scores , 2004 .

[38]  Scott E. Maxwell,et al.  Designing Experiments and Analyzing Data: A Model Comparison Perspective , 1990 .

[39]  Herman Wold,et al.  Soft modelling: The Basic Design and Some Extensions , 1982 .

[40]  Straub,et al.  Editor's Comments: An Update and Extension to SEM Guidelines for Administrative and Social Science Research , 2011 .

[41]  William Lewis,et al.  Research Note - Statistical Power in Analyzing Interaction Effects: Questioning the Advantage of PLS with Product Indicators , 2007, Inf. Syst. Res..

[42]  D. W. Zimmerman,et al.  Correction for Attenuation With Biased Reliability Estimates and Correlated Errors in Populations and Samples , 2007 .

[43]  C. Saunders,et al.  Editor's comments: PLS: a silver bullet? , 2006 .

[44]  Bill Wilson,et al.  In a relationship , 2013 .

[45]  David F. Larcker,et al.  Structural Equation Models with Unobservable Variables and Measurement Error: Algebra and Statistics: , 1981 .

[46]  Karl G. Jöreskog,et al.  Lisrel 8: User's Reference Guide , 1997 .

[47]  Eric P. Charles,et al.  The correction for attenuation due to measurement error: clarifying concepts and creating confidence sets. , 2005, Psychological methods.

[48]  Todd D. Little,et al.  A Non-arbitrary Method of Identifying and Scaling Latent Variables in SEM and MACS Models , 2006 .

[49]  Wynne W. Chin,et al.  Structural equation modeling analysis with small samples using partial least squares , 1999 .

[50]  K. Jöreskog,et al.  Intraclass Reliability Estimates: Testing Structural Assumptions , 1974 .

[51]  R E Millsap,et al.  Confirmatory Measurement Model Comparisons Using Latent Means. , 1991, Multivariate behavioral research.

[52]  Xitao Fan,et al.  TEACHER'S CORNER: Using SAS for Monte Carlo Simulation Research in SEM , 2005 .