Where’s the difficulty in standardized reading tests: The passage or the question?

The purpose of this investigation was to examine the extent to which item and text characteristics predict item difficulty on the comprehension portion of the Gates-MacGinitie Reading Tests for the 7th–9th and 10th–12th grade levels. Detailed item-based analyses were performed on 192 comprehension questions on the basis of the cognitive processing model framework proposed by Embretson and colleagues (Embretson & Wetzel, 1987). Item difficulty was analyzed in terms of various passage features (e.g., word frequency and number of propositions) and individual-question characteristics (e.g., abstractness and degree of inferential processing), using hierarchical linear modeling. The results indicated that the difficulty of the items in the test for the 7th–9th grade level is primarily influenced by text features—in particular, vocabulary difficulty—whereas the difficulty of the items in the test for the 10th–12th grade level is less systematically influenced by text features.

[1]  W D Marslen-Wilson,et al.  Sentence Perception as an Interactive Parallel Process , 1975, Science.

[2]  Danielle S. McNamara,et al.  Influence of Question Format and Text Availability on the Assessment of Expository Text Comprehension , 2007 .

[3]  M. Just,et al.  From the SelectedWorks of Marcel Adam Just 1992 A capacity theory of comprehension : Individual differences in working memory , 2017 .

[4]  R. Freedle.  New directions in discourse processing , 1980 .

[5]  S. Embretson,et al.  Component Latent Trait Models for Paragraph Comprehension Tests , 1987 .

[6]  Robert F. Lorch,et al.  The effects of readers’ goals on inference generation and memory for texts , 2001, Memory & cognition.

[7]  Richard C. Anderson.  How to Construct Achievement Tests to Assess Comprehension , 1972 .

[8]  Robert J. Mislevy,et al.  PROBABILITY‐BASED INFERENCE IN COGNITIVE DIAGNOSIS , 1994 .

[9]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[10]  Steven A. Stahl,et al.  Children's reading comprehension and assessment , 2005 .

[11]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[12]  David J. Weiss,et al.  APPLICATION OF COMPUTERIZED ADAPTIVE TESTING TO EDUCATIONAL PROBLEMS , 1984 .

[13]  W. Kintsch.  The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[14]  Peter B. Mosenthal,et al.  Understanding the strategies of document literacy and their conditions of use. , 1996 .

[15]  Jack M. Fletcher,et al.  Measuring Reading Comprehension , 2006 .

[16]  Jeanne D. Day,et al.  Strategy use on standardized reading comprehension tests. , 1996 .

[17]  Gary W. Phillips,et al.  Technical Issues in Large-Scale Performance Assessment. , 1996 .

[18]  Arthur C. Graesser,et al.  Coh-Metrix: Analysis of text on cohesion and language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[19]  P. David Pearson,et al.  Principles for Classroom Comprehension Assessment , 1988 .

[20]  S. Embretson.  A cognitive design system approach to generating valid tests : Application to abstract reasoning , 1998 .

[21]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[22]  R. Mislevy.  Evidence and inference in educational assessment , 1994 .

[23]  Jaekyung Lee,et al.  Evaluating the effectiveness of instructional resource allocation and use: IRT and HLM analysis of NAEP teacher survey and student assessment data , 2004 .

[24]  Brent Bridgeman,et al.  Relationships Among Multiple-Choice and Open-Ended Analytical Questions , 1993 .

[25]  Brenda Hannon,et al.  Using working memory theory to investigate the construct validity of multiple-choice reading comprehension tests such as the SAT. , 2001, Journal of experimental psychology. General.

[26]  Michael B. W. Wolfe,et al.  Relations Between Adolescents' Text Processing and Reasoning , 2005 .

[27]  P. David Pearson,et al.  The Assessment of Reading Comprehension: A Review of Practices—Past, Present, and Future , 2005 .

[28]  Tobias Richter,et al.  What Is Wrong With ANOVA and Multiple Regression? Analyzing Sentence Reading Times With Hierarchical Linear Models , 2006 .

[29]  Danielle S. McNamara,et al.  A multidimensional framework to evaluate reading assessment tools. , 2007 .

[30]  Charles A. Perfetti,et al.  Phonemic knowledge and learning to read are reciprocal: A longitudinal study of first grade children. , 1987 .

[31]  Eugene Charniak,et al.  A Maximum-Entropy-Inspired Parser , 2000, ANLP.

[32]  Stuart Katz,et al.  Answering Reading Comprehension Items without Passages on the Sat when Items are Quasi-Randomized , 1991 .

[33]  Samuel Messick,et al.  Standards-Based Score Interpretation: Establishing Valid Grounds for Valid Inferences. Research Report. , 1994 .

[34]  M. Daneman,et al.  A new tool for measuring and understanding individual differences in the component processes of reading comprehension , 2001 .

[35]  Jim J. Manhart,et al.  Factor Analytic Methods for Determining Whether Multiple-Choice and Constructed-Response Tests Measure the Same Construct. , 1996 .

[36]  A. Collins,et al.  A schema-theoretic view of reading , 1977 .

[37]  R. Almond,et al.  Making Sense of Data From Complex Assessments , 2002 .

[38]  Robert J. Sternberg,et al.  Are We Reading Too Much into Reading Comprehension Tests , 1991 .

[39]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .