Sources of Difference in Reliability: Identifying Sources of Difference in Reliability in Content Analysis of Online Asynchronous Discussions.

This paper reports on a case study which identifies and illustrates sources of difference in agreement in relation to reliability in a context of quantitative content analysis of a transcript of an online asynchronous discussion (OAD). Transcripts of 10 students in a month-long online asynchronous discussion were coded by two coders using an instrument with two categories, five processes, and 19 indicators of Problem Formulation and Resolution (PFR). Sources of difference were identified in relation to: coders; tasks; and students. Reliability values were calculated at the levels of categories, processes, and indicators. At the most detailed level of coding on the basis of the indicator, findings revealed that the overall level of reliability between coders was .591 when measured with Cohen’s kappa. The difference between tasks at the same level ranged from .349 to .664, and the difference between participants ranged from .390 to .907. Implications for training and research are discussed.

[1]  W. James Potter,et al.  Rethinking validity and reliability in content analysis , 1999 .

[2]  Steven E. Stemler.  Practical Assessment, Research, and Evaluation Practical Assessment, Research, and Evaluation A Comparison of Consensus, Consistency, and Measurement A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability Approaches to Estimating Interrater Reliabilit , 2022 .

[3]  Colin Seymour-Ure,et al.  Content Analysis in Communication Research. , 1972 .

[4]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[5]  M. Lombard,et al.  Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability , 2002 .

[6]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[7]  T. Anderson,et al.  Online Social Interchange, Discord, and Knowledge Construction , 1998 .

[8]  Daniel C. A. Hillman,et al.  A New Method for Analyzing Patterns of Interaction , 1999 .

[9]  Elizabeth Murphy,et al.  Promoting construct validity in instruments for the analysis of transcripts of online asynchronous discussions , 2004 .

[10]  Liam Rourke,et al.  Validity in quantitative content analysis , 2004 .

[11]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[12]  Dale J. Prediger,et al.  Coefficient Kappa: Some Uses, Misuses, and Alternatives , 1981 .

[13]  D. Garrison,et al.  Methodological Issues in the Content Analysis of Computer Conference Transcripts , 2007 .

[14]  M. Banerjee,et al.  Beyond kappa: A review of interrater agreement measures , 1999 .

[15]  Tarald O. Kvålseth,et al.  Note on Cohen's Kappa , 1989 .

[16]  R. Kolbe,et al.  Content-Analysis Research: An Examination of Applications with Directives for Improving Research Reliability and Objectivity , 1991 .

[17]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[18]  Michael W. Singletary.  Mass Communication Research: Contemporary Methods and Applications , 1993 .

[19]  Roel Popping,et al.  On Agreement Indices for Nominal Data , 1988 .

[20]  D. Garrison,et al.  Critical thinking, cognitive presence, and computer conferencing in distance education , 2001 .

[21]  D. Weiss,et al.  Interrater reliability and agreement of subjective judgments , 1975 .

[22]  W. D. Perreault,et al.  Reliability of Nominal Data Based on Qualitative Judgments , 1989 .