The Assumption of a Reliable Instrument and Other Pitfalls to Avoid When Considering the Reliability of Data

The purpose of this article is to help researchers avoid common pitfalls associated with reliability including incorrectly assuming that (a) measurement error always attenuates observed score correlations, (b) different sources of measurement error originate from the same source, and (c) reliability is a function of instrumentation. To accomplish our purpose, we first describe what reliability is and why researchers should care about it with focus on its impact on effect sizes. Second, we review how reliability is assessed with comment on the consequences of cumulative measurement error. Third, we consider how researchers can use reliability generalization as a prescriptive method when designing their research studies to form hypotheses about whether or not reliability estimates will be acceptable given their sample and testing conditions. Finally, we discuss options that researchers may consider when faced with analyzing unreliable data.

[1]  Kate E Decleene,et al.  Publication Manual of the American Psychological Association , 2011 .

[2]  Bruce Thompson,et al.  Score Reliability: A Retrospective Look Back at 12 Years of Reliability Generalization Studies , 2011 .

[3]  Rui Yao,et al.  Publication Manual of the American Psychological Association , 2011 .

[4]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[5]  Pere J. Ferrando,et al.  Two SPSS programs for interpreting multiple regression results , 2010, Behavior research methods.

[6]  Bruce Thompson,et al.  Demonstration of How Score Reliability is Integrated Into SEM and How Reliability Affects All Statistical Analyses , 2010 .

[7]  A. Zellner,et al.  INTRODUCTION TO MEASUREMENT WITH THEORY , 2009, Macroeconomic Dynamics.

[8]  B. Thompson,et al.  Matrix Summaries Improve Research Reports: Secondary Analyses Using Published Literature , 2009 .

[9]  David Trafimow,et al.  Potential performance theory (PPT): Describing a methodology for analyzing task performance , 2009, Behavior research methods.

[10]  Robert M. Capraro,et al.  Reporting Practices in Quantitative Teacher Education Research: One Look at the Evidence Cited in the AERA Panel Report , 2008 .

[11]  D. W. Zimmerman,et al.  Correction for Attenuation With Biased Reliability Estimates and Correlated Errors in Populations and Samples , 2007 .

[12]  Christopher S. Miller,et al.  Substance Use Scales of the Minnesota Multiphasic Personality Inventory , 2007 .

[13]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[14]  Michael C. Rodriguez,et al.  Meta-analysis of coefficient alpha. , 2006, Psychological methods.

[15]  Pamela A. Moss,et al.  Standards for Reporting on Empirical Social Science Research in AERA Publications American Educational Research Association , 2006 .

[16]  Debra Wetcher-Hendricks.  Adjustments to the correction for attenuation. , 2006, Psychological methods.

[17]  Leslie R. Odom,et al.  A Reliability Generalization Study of the Self-Description Questionnaire , 2006 .

[18]  A. Bandura.  GUIDE FOR CONSTRUCTING SELF-EFFICACY SCALES , 2006 .

[19]  Eric P. Charles,et al.  The correction for attenuation due to measurement error: clarifying concepts and creating confidence sets. , 2005, Psychological methods.

[20]  A. Onwuegbuzie,et al.  A Proposed New “What if Reliability” Analysis for Assessing the Statistical Significance of Bivariate Relationships , 2005 .

[21]  Deepa Marat,et al.  Assessing Mathematics Self-Efficacy of Diverse Students from Secondary Schools in Auckland: Implications for Academic Achievement. , 2005 .

[22]  L. Cronbach,et al.  My Current Thoughts on Coefficient Alpha and Successor Procedures , 2004 .

[23]  A. Shields,et al.  A Reliability Induction and Reliability Generalization Study of the Cage Questionnaire , 2004 .

[24]  Steven E. Stemler.  Practical Assessment, Research, and Evaluation Practical Assessment, Research, and Evaluation A Comparison of Consensus, Consistency, and Measurement A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability Approaches to Estimating Interrater Reliabilit , 2022 .

[25]  T. Bruce.  Guidelines for Authors Reporting Score Reliability Estimates , 2003 .

[26]  Bruce Thompson,et al.  A brief introduction to generalizability theory. , 2003 .

[27]  D. Dimitrov.  Reliability: Arguments for Multiple Perspectives and Potential Problems with Generalization across Studies , 2002 .

[28]  Robin K. Henson,et al.  Reliability Generalization: Moving toward Improved Understanding and Use of Score Reliability , 2002 .

[29]  小嶋 雅代,et al.  日本語版 Beck Depression Inventory-II (BDI-II) の開発 , 2002 .

[30]  Jason W. Osbourne,et al.  Four Assumptions of Multiple Regression That Researchers Should Always Test. , 2002 .

[31]  R. Henson.  Understanding Internal Consistency Reliability Estimates: A Conceptual Primer on Coefficient Alpha , 2001 .

[32]  Bruce Thompson,et al.  Statistical Techniques Employed in AERJ and JCP Articles from 1988 to 1997: A Methodological Review , 2001 .

[33]  B. Thompson,et al.  Sample Compositions and Variabilities in Published Studies versus Those in Test Manuals: Validity of Score Reliability Inductions , 2000 .

[34]  T. Hogan,et al.  Reliability Methods: A Note on the Frequency of Use of Various Types , 2000 .

[35]  Bruce Thompson,et al.  Psychometrics is Datametrics: the Test is not Reliable , 2000 .

[36]  B. Thompson.  Ten commandments of structural equation modeling. , 2000 .

[37]  Paul R. Yarnold,et al.  Reading and understanding MORE multivariate statistics. , 2000 .

[38]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[39]  W. James Popham,et al.  Modern Educational Measurement: Practical Guidelines for Educational Leaders , 1999 .

[40]  Pajares,et al.  Self-Efficacy, Motivation Constructs, and Mathematics Performance of Entering Middle School Students. , 1999, Contemporary educational psychology.

[41]  Johanna E. Nilsson,et al.  Practices Regarding Reporting of Reliability Coefficients: A Review of Three Journals , 1999 .

[42]  David J. A. Dozois,et al.  A psychometric evaluation of the Beck Depression inventory-II , 1998 .

[43]  T. Vacha-Haase,et al.  Reliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies , 1998 .

[44]  S. Urbina,et al.  Psychological testing, 7th ed. , 1997 .

[45]  Robert Rosenthal,et al.  WRITING META-ANALYTIC REVIEWS , 1995 .

[46]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[47]  P. Pintrich.  A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). , 1991 .

[48]  Herbert W. Marsh,et al.  Age and sex effects in multiple dimensions of self-concept: Preadolescence to early adulthood. , 1989 .

[49]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[50]  E. Pedhazur.  Multiple Regression in Behavioral Research: Explanation and Prediction , 1982 .

[51]  W. Grove.  Statistical Methods for Rates and Proportions, 2nd ed , 1981 .

[52]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[53]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[54]  John E. Hunter,et al.  Development of a general solution to the problem of validity generalization. , 1977 .

[55]  The theory of test validity and correlated errors of measurement , 1977 .

[56]  J. Fleiss,et al.  Statistical methods for rates and proportions , 1973 .

[57]  Y. Morrison.  presented at the Annual Meeting of the , 1970 .

[58]  J. Keats,et al.  Test theory. , 1967, Annual review of psychology.

[59]  H. Harman.  Modern factor analysis , 1961 .

[60]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[61]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[62]  L. L. Thurstone,et al.  The correction for attenuation. , 1931 .