Developing Multidimensional Likert Scales Using Item Factor Analysis

This study compares the performance of two approaches in analysing four-point Likert rating scales with a factorial model: the classical factor analysis (FA) and the item factor analysis (IFA). For FA, maximum likelihood and weighted least squares estimations using Pearson correlation matrices among items are compared. For IFA, diagonally weighted least squares and unweighted least squares estimations using items polychoric correlation matrices are compared. Two hundred and ten conditions were simulated in a Monte Carlo study considering: one to three factor structures (either, independent and correlated in two levels), medium or low quality of items, three different levels of item asymmetry and five sample sizes. Results showed that IFA procedures achieve equivalent and accurate parameter estimates; in contrast, FA procedures yielded biased parameter estimates. Therefore, we do not recommend classical FA under the conditions considered. Minimum requirements for achieving accurate results using IFA procedures are discussed.

[1]  R. Garland.  The Mid-Point on a Rating Scale: Is it Desirable? , 1991 .

[2]  André Beauducel,et al.  On the Performance of Maximum Likelihood Versus Means and Variance Adjusted Weighted Least Squares Estimation in CFA , 2006 .

[3]  Alberto Maydeu-Olivares,et al.  Estimation of IRT graded response models: limited versus full information methods. , 2009, Psychological methods.

[4]  Leonard R. Sussman,et al.  Nominal, Ordinal, Interval, and Ratio Typologies are Misleading , 1993 .

[5]  H. Marsh,et al.  The use of item parcels in structural equation modelling: non-normal data and small sample sizes. , 2004, The British journal of mathematical and statistical psychology.

[6]  M. Browne.  Asymptotically distribution-free methods for the analysis of covariance structures. , 1984, The British journal of mathematical and statistical psychology.

[7]  J. Gaito.  Measurement scales and statistics: Resurgence of an old misconception. , 1980 .

[8]  Robert F. DeVellis,et al.  Scale Development: Theory and Applications. , 1992 .

[9]  Q. Raaijmakers,et al.  Adolescents' midpoint responses on Likert-type scale items: Neutral or missing values? , 2000 .

[10]  B. Muthén,et al.  Dichotomous Factor Analysis of Symptom Data , 1989 .

[11]  F. Holgado-Tello,et al.  Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables , 2008 .

[12]  C. Ferguson.  An effect size primer: A guide for clinicians and researchers. , 2009 .

[13]  Robert E. Widing,et al.  Customer evaluation of retail salespeople utilizing the SOCO scale: A replication, extension, and application , 1991 .

[14]  Anders Christoffersson,et al.  Factor analysis of dichotomized variables , 1975 .

[15]  D. Andrich.  A rating formulation for ordered response categories , 1978 .

[16]  John T. Kulas,et al.  Middle Response Functioning in Likert-responses to Personality Items , 2008 .

[17]  J. Jacoby,et al.  Is There an Optimal Number of Alternatives for Likert Scale Items? Study I: Reliability and Validity , 1971 .

[18]  G. Hancock,et al.  The effect of scale manipulations on validity: targetting frequency rating scales for anticipated performance levels. , 1991, Applied ergonomics.

[19]  H. Kraemer,et al.  Summated Rating Scale Construction An Introduction , 2009 .

[20]  Gardner Murphy,et al.  A Simple and Reliable Method of Scoring the Thurstone Attitude Scales , 1934 .

[21]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[22]  Rocco J. Perla,et al.  Ten Common Misunderstandings, Misconceptions, Persistent Myths and Urban Legends about Likert Scales and Likert Response Formats and their Antidotes , 2007 .

[23]  Two-step weighted least squares factor analysis of dichotomized variables , 1977 .

[24]  Michael R. Harwell,et al.  Monte Carlo Studies in Item Response Theory , 1996 .

[25]  Christine DiStefano,et al.  The Impact of Categorization With Confirmatory Factor Analysis , 2002 .

[26]  Conor V. Dolan,et al.  Factor analysis of variables with 2, 3, 5, and 7 response categories: A comparison of categorical variable estimators using simulated data , 1994 .

[27]  G. Norman.  Likert scales, levels of measurement and the “laws” of statistics , 2010, Advances in health sciences education : theory and practice.

[28]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[29]  Alberto Maydeu-Olivares,et al.  Factor Analysis with Ordinal Indicators: A Monte Carlo Study Comparing DWLS and ULS Estimation , 2009 .

[30]  B. Muthén.  Contributions to factor analysis of dichotomous variables , 1978 .

[31]  T. E. Dinero.  Scale development. , 1996, Journal of health & social policy.

[32]  Eli P. Cox,et al.  The Optimal Number of Response Alternatives for a Scale: A Review , 1980 .

[33]  Karl G. Jöreskog,et al.  Lisrel 8: User's Reference Guide , 1997 .

[34]  W. R. Garner,et al.  Rating scales, discriminability, and information transmission. , 1960, Psychological review.

[35]  S. Jamieson.  Likert scales: how to (ab)use them , 2004, Medical education.

[36]  A. Boomsma,et al.  Robustness Studies in Covariance Structure Modeling , 1998 .

[37]  B. Muthén.  A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators , 1984 .

[38]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[39]  Ira H. Bernstein,et al.  Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. , 1989 .

[40]  D. Flora,et al.  An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. , 2004, Psychological methods.

[41]  Lei Chang.  A Psychometric Evaluation of 4-Point and 6-Point Likert-Type Scales in Relation to Reliability and Validity , 1994 .

[42]  K. Jöreskog,et al.  Confirmatory Factor Analysis of Ordinal Variables With Misspecified Models , 2010 .

[43]  A. Colman,et al.  Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. , 2000, Acta psychologica.

[44]  Frederic M. Lord,et al.  On the Statistical Treatment of Football Numbers. , 1953 .

[45]  Duane T. Wegener,et al.  Evaluating the use of exploratory factor analysis in psychological research. , 1999 .

[46]  Roderick P. McDonald,et al.  Linear Versus Models in Item Response Theory , 1982 .

[47]  J. Michell.  The psychometricians' fallacy: too clever by half? , 2009, The British journal of mathematical and statistical psychology.

[48]  B. Muthén,et al.  Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes , 1997 .

[49]  Kristopher J Preacher,et al.  Item factor analysis: current approaches and future directions. , 2007, Psychological methods.

[50]  Pere J. Ferrando,et al.  FACTOR: A computer program to fit the exploratory factor analysis model , 2006, Behavior research methods.

[51]  V. Savalei,et al.  When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. , 2012, Psychological methods.

[52]  V. Savalei,et al.  The performance of robust test statistics with categorical data. , 2013, The British journal of mathematical and statistical psychology.

[53]  Edward E. Rigdon,et al.  The Performance of the Polychoric Correlation Coefficient and Selected Fitting Functions in Confirmatory Factor Analysis with Ordinal Data , 1991 .

[54]  B. Espejo,et al.  Testing the middle response categories "Not sure", "In between" and "?" in polytomous items , 2003 .

[55]  Francisco Pablo Holgado Tello,et al.  Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables , 2010 .