Establishing reliable procedures for rating ell students’ reading comprehension using oral retellings

Abstract Oral retellings are strongly recommended as a way to measure reading comprehension for second language learners (Bernhardt, 1985, 1990, 1991). However, the reliability of such ratings is a matter of concern for a variety of reasons (Aiken, 1996; Cooper, 1981; Saal, Downey, & Lahey, 1980). The purpose of this study was to establish reliable rating procedures to assess the reading comprehension of college‐age second language learners using oral retellings. A three‐facet (passage, rater, and rating occasion) generalizability study was conducted using oral retellings of expository passages. Retellings of three passages provided by 24 ELL students were rated by two trained raters on two occasions. The largest sources of error variance identified were the passages and the student‐by‐passage interaction. Results showed that the most important step that could be taken to reduce measurement error and increase generalizability would be to use at least four and preferably six passages to assess ELL students’ reading comprehension. The gain in generalizability from using additional raters or rating occasions was minimal.

[1]  Knowing Literacy , 1998 .

[2]  Robert L. Brennan,et al.  Generalizability of Performance Assessments , 2005 .

[3]  Lesley Mandel Morrow,et al.  Effects of Structural Guidance in Story Retelling on Children's Dictation of Original Stories , 1986 .

[4]  J. Parkes,et al.  The Role of Transfer in the Variability of Performance Assessment Scores , 2001 .

[5]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[6]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[7]  Elizabeth B. Bernhardt Testing Foreign Language Reading Comprehension: The Immediate Recall Protocol. , 1983 .

[8]  Richard T. Vacca,et al.  Content Area Reading , 1981 .

[9]  D. Sengupta Linear models , 2003 .

[10]  E. Fry A Readability Formula for Short Passages. , 1990 .

[11]  Michael T. Kane,et al.  Inferences about variance components and reliability-generalizability coefficients in the absence of random sampling. , 2002 .

[12]  Geoffrey R. Norman,et al.  Performance-Based Assessment: Lessons From the Health Professions , 1995 .

[13]  Edward B. Fry,et al.  Fry's Readability Graph: Clarifications, Validity, and Extension to Level 17. , 1977 .

[14]  Karen K. Wixson,et al.  Assessment and Instruction of Reading Disability: An Interactive Approach , 1991 .

[15]  Elizabeth B. Bernhardt Reading Development in a Second Language: Theoretical, Empirical and Classroom Perspectives , 1991 .

[16]  Sheila W. Valencia Literacy Portfolios in Action. , 1998 .

[17]  Robert B. Ruddell,et al.  Teaching Children To Read and Write: Becoming an Influential Teacher. Second Edition. , 1995 .

[18]  M. Strube Reliability and generalizability theory. , 2000 .

[19]  L. Morrow Reading and Retelling Stories: Strategies for Emergent Readers. , 1985 .

[20]  Peter H. Johnston,et al.  Reading Comprehension Assessment: A Cognitive Basis , 1983 .

[21]  Paul R. Yarnold,et al.  Reading and understanding MORE multivariate statistics. , 2000 .

[22]  Bruce Thompson,et al.  A brief introduction to generalizability theory. , 2003 .

[23]  Albert C. Oosterhof,et al.  Developing and using classroom assessments , 1996 .

[24]  T. Contents,et al.  Making ends meet , 2020, Nature.

[25]  R. Downey,et al.  Rating the ratings: Assessing the psychometric quality of rating data , 1980 .

[26]  R. L. Ebel,et al.  Estimation of the reliability of ratings , 1951 .

[27]  Nigel O'Brian,et al.  Generalizability Theory I , 2003 .

[28]  Gail E. Tompkins Literacy for the 21st Century: A Balanced Approach , 1996 .

[29]  L. Lein,et al.  Making Ends Meet , 1997 .

[30]  Robert L. Linn,et al.  Performance Assessment: Policy Promises and Technical Measurement Standards , 1994 .

[31]  R. Brennan Elements of generalizability theory , 1983 .

[32]  Noreen M. Webb,et al.  Using Generalizability Theory in Counseling and Development. , 1988 .