Multiple-choice versus open-ended response formats of reading test items: A two-dimensional IRT analysis

The dimensionality of a reading comprehension assessment with non-stem equivalent multiplechoice (MC) items and open-ended (OE) items was analyzed with German test data of 8523 9 th -graders. We found that a two-dimensional IRT model with within-item multidimensionality, where MC and OE items load on a general latent dimension and OE items additionally load on a nested latent dimension, had a superior fit compared to an unidimensional model (p ≤ .05). Correlations between general cognitive abilities, orthography and vocabulary and the general latent dimension were significantly higher than with the nested latent dimension (p ≤ .05). Drawing back on experimental studies on the effect of item format on reading processes, we suppose that the general latent dimension measures abilities necessary to master basic reading processes and the nested latent dimension captures abilities necessary to master higher reading processes. Including gender, language spoken at home, and school track as predictors in latent regression models showed that the well known advantage of girls and mother-tongue students is found only for the nested latent dimension.

[1]  J. B. Wyman,et al.  What is reading ability , 1921 .

[2]  James I. Brown.  The Nelson-Denny Reading Test. , 1960 .

[3]  E. Gibson.  Learning to read , 1965, Science.

[4]  S. Jay Samuels,et al.  Toward a theory of automatic information processing in reading , 1974 .

[5]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[6]  Jerry L. Johns.  Do Comprehension Items Really Test Reading? Sometimes!. , 1978 .

[7]  Elana Shohamy.  Does the testing method make a difference? The case of reading comprehension , 1984 .

[8]  T. A. Warm.  Weighted likelihood estimation of ability in item response theory , 1989 .

[9]  K. Tatsuoka,et al.  Open-Ended Versus Multiple-Choice Response Formats—It Does Make a Difference for Diagnostic Purposes , 1987 .

[10]  B. Davey.  Postpassage Questions: Task and Reader Effects on Comprehension and Metacomprehension Processes , 1987 .

[11]  B. Davey.  The Nature of Response Errors for Good and Poor Readers When Permitted to Reinspect Text During Question-Answering , 1988 .

[12]  W. Schneider,et al.  Domain-Specific Knowledge and Memory Performance: A Comparison of High- and Low-Aptitude Children , 1989 .

[13]  Mark D. Reckase,et al.  The Discriminating Power of Items That Measure More Than One Dimension , 1991 .

[14]  Susan E. Embretson,et al.  A multidimensional latent trait model for measuring learning and change , 1991 .

[15]  Randy Elliot Bennett,et al.  Equivalence of Free-Response and Multiple-Choice Items , 1991 .

[16]  G. D. N. Worswick,et al.  Unemployment: A Problem of Policy: CONCEPTS AND MEASUREMENTS , 1991 .

[17]  Richard M. Luecht,et al.  Unidimensional Calibrations and Interpretations of Composite Traits for Multidimensional Tests , 1992 .

[18]  C. Perfetti.  The Representation Problem in Reading Acquisition , 1992 .

[19]  Karen Draney,et al.  Objective measurement : theory into practice , 1992 .

[20]  T. C. Oshima,et al.  Multidimensionality and Item Bias in Item Response Theory , 1992 .

[21]  Gregory Camilli,et al.  A Conceptual Analysis of Differential Item Functioning in Terms of a Multidimensional Item Response Model , 1992 .

[22]  Howard Wainer,et al.  COMBINING MULTIPLE-CHOICE AND CONSTRUCTED RESPONSE TEST SCORES: TOWARD A MARXIST THEORY OF TEST CONSTRUCTION , 1992 .

[23]  Connie K. Varnhagen,et al.  Structural components of reading time and recall for sentences in narratives : exploring changes with age and reading ability , 1992 .

[24]  Terry A. Ackerman.  A Didactic Explanation of Item Bias, Item Impact, and Item Validity from a Multidimensional Perspective , 1992 .

[25]  D. Treiman,et al.  A standard international socio-economic index of occupational status , 1992 .

[26]  W Kintsch,et al.  Writing quality, reading skills, and domain knowledge as factors in text comprehension. , 1993, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[27]  Darlene F. Wolf,et al.  A Comparison of Assessment Tasks Used to Measure FL Reading Comprehension , 1993 .

[28]  Robert van Krieken.  Construct Validation of Question Formats for Dutch Central Examinations in Foreign Language Reading Comprehension , 1993 .

[29]  H. Wainer,et al.  Are Tests Comprising Both Multiple‐Choice and Free‐Response Items Necessarily Less Unidimensional Than Multiple‐Choice Tests?An Analysis of Two Tests , 1994 .

[30]  R. Sternberg,et al.  The Road Not Taken , 1994, Journal of learning disabilities.

[31]  Jeanne D. Day,et al.  Strategy use on standardized reading comprehension tests. , 1996 .

[32]  James F. Voss,et al.  Learning From History Text: The Interaction of Knowledge and Comprehension Skill with Text Structure , 1996 .

[33]  Raymond J. Adams,et al.  The Multidimensional Random Coefficients Multinomial Logit Model , 1997 .

[34]  S. Jay Samuels,et al.  THE IMPORTANCE OF AUTOMATICITY FOR DEVELOPING EXPERTISE IN READING , 1997 .

[35]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[36]  Catherine E. Snow,et al.  Preventing reading difficulties in young children , 1998 .

[37]  Margaret Wu,et al.  ACER conquest: generalised item response modelling software , 1998 .

[38]  Kadriye Ercikan,et al.  Calibration and Scoring of Tests With Multiple-Choice and Constructed-Response Item Types , 1998 .

[39]  L. Katz,et al.  Subtypes of reading disability: Variability around a phonological core. , 1998 .

[40]  Albert Satorra,et al.  A scaled difference chi-square test statistic for moment structure analysis , 1999 .

[41]  W. Becker,et al.  The Relationship between Multiple Choice and Essay Response Questions in Assessing Economics Understanding , 1999 .

[42]  J. Alderson.  Assessing Reading: Acknowledgements , 2000 .

[43]  Stuart Katz,et al.  Answering Reading Comprehension Items without the Passages on the SAT–I , 1999 .

[44]  H. Lee Swanson,et al.  Cognitive Processing of Low Achievers and Children with Reading Disabilities: A Selective Meta-Analytic Review of the Published Literature , 2000 .

[45]  Dorothy V. M. Bishop,et al.  Speech and Language Impairments in Children : Causes, Characteristics, Intervention and Outcome , 2014 .

[46]  R. P. McDonald,et al.  A Basis for Multidimensional Item Response Theory , 2000 .

[47]  S. Natasha Beretvas,et al.  An empirical investigation demonstrating the multidimensional DIF paradigm: A cognitive explanation for DIF , 2001 .

[48]  David K. Dickinson,et al.  Handbook of early literacy research , 2001 .

[49]  C. Artelt,et al.  Lesekompetenz: Testkonzeption und Ergebnisse , 2001 .

[50]  Brenda Hannon,et al.  Using working memory theory to investigate the construct validity of multiple-choice reading comprehension tests such as the SAT. , 2001, Journal of experimental psychology. General.

[51]  Petra Stanat,et al.  Leseleistungen deutscher Schülerinnen und Schüler im internationalen Vergleich (PISA) , 2002 .

[52]  Jack M. Fletcher,et al.  Validity of IQ-Discrepancy Classifications of Reading Disabilities: A Meta-Analysis , 2002 .

[53]  M. Kobayashi.  Method effects on reading comprehension test performance: text organization and response format , 2002 .

[54]  Derek C. Briggs,et al.  An introduction to multidimensional measurement using Rasch models. , 2003, Journal of applied measurement.

[55]  A. Graesser,et al.  Handbook of discourse processes , 2003 .

[56]  González y Ortiz,et al.  Knowledge and Skills for Life, first results from the OECD Programme for International Student Assessment (PISA) 2000 , 2003 .

[57]  Michael C. Rodriguez.  Construct Equivalence of Multiple-Choice and Constructed-Response Items: A Random Effects Synthesis of Correlations , 2003 .

[58]  K. Kubinger.  Psychological Test Calibration Using the Rasch Model—Some Critical Suggestions on Traditional Approaches , 2005 .

[59]  Tihomir Asparouhov,et al.  Multivariate Statistical Modeling with Survey Data , 2005 .

[60]  Gene Ouellette,et al.  What's Meaning Got to Do With It: The Role of Vocabulary in Word Reading and Reading Comprehension , 2006 .

[61]  Gayle S. Christensen,et al.  Where immigrant students succeed: A comparative review of performance and engagement in PISA 2003 , 2006 .

[62]  Assessment Framework and Specifications (2nd Edition). PIRLS 2006. , 2006 .

[63]  Andreas Schleicher.  Where immigrant students succeed: a comparative review of performance and engagement in PISA 2003 1 , 2006 .

[64]  Danielle S. McNamara,et al.  Influence of Question Format and Text Availability on the Assessment of Expository Text Comprehension , 2007 .

[65]  Representation of Competencies in Multidimensional IRT Models with Within-Item and Between-Item Multidimensionality , 2008 .

[66]  George K. Georgiou,et al.  Predictors of word decoding and reading fluency across languages varying in orthographic consistency. , 2008 .

[67]  Eckhard Klieme,et al.  Unterricht und Kompetenzerwerb in Deutsch und Englisch. Ergebnisse der DESI-Studie , 2008 .

[68]  Nicole J. Conrad.  From Reading to Spelling and Spelling to Reading: Transfer Goes both Ways. , 2008 .

[69]  Sooyeon Kim,et al.  EQUATING OF MIXED-FORMAT TESTS IN LARGE-SCALE ASSESSMENTS , 2008 .

[70]  Johannes Hartig,et al.  Multidimensional IRT models for the assessment of competencies , 2009 .

[71]  Petra Stanat,et al.  Schülerinnen und Schüler mit Migrationshintergrund , 2010 .