The TextEvaluator Tool

This article describes TextEvaluator, a comprehensive text-analysis system designed to help teachers, textbook publishers, test developers, and literacy researchers select reading materials that are consistent with the text-complexity goals outlined in the Common Core State Standards. Three particular aspects of the TextEvaluator measurement approach are highlighted: (1) attending to relevant reader and task considerations, (2) expanding construct coverage beyond the two dimensions of text variation traditionally assessed by readability metrics, and (3) addressing two potential threats to tool validity: genre bias and blueprint bias. We argue that systems that are attentive to these particular measurement issues may be more effective at helping users achieve a key goal of the new Standards: ensuring that students are challenged to read texts at steadily increasing complexity levels as they progress through school, so that all students acquire the advanced reading skills needed for success in college and careers.

[1]  M. Just,et al.  The psychology of reading and language comprehension , 1986 .

[2]  Michael Flor,et al.  A Two-Stage Approach for Generating Unbiased Estimates of Text Complexity , 2013 .

[3]  K. Sheehan,et al.  When Do Standard Approaches for Measuring Vocabulary Difficulty , Syntactic Complexity and Referential Cohesion Yield Biased Estimates of Text Difficulty ? , 2008 .

[4]  飯島 周.  「会話の文法」に関する一考察 : Longman Grammar of Spoken and Written Englishの場合 , 1999 .

[5]  Michael Wilson.  MRC Psycholinguistic Database , 2001 .

[6]  John Sabatini,et al.  Differences in Text Structure and Its Implications for Assessment of Struggling Readers , 2006 .

[7]  D. Borsboom.  Educational Measurement (4th ed.) , 2009 .

[8]  T. Landauer,et al.  Pearson's Text Complexity Measure White Paper Pearson's Text Complexity Measure , 2022 .

[9]  Arthur C. Graesser,et al.  Coh-Metrix: Analysis of text on cohesion and language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[10]  Douglas Biber,et al.  Variation across speech and writing: Methodology , 1988 .

[11]  Yoko Futagi,et al.  Generating Automated Text Complexity Classifications That Are Aligned with Targeted Text Complexity Standards. Research Report. ETS RR-10-28. , 2010 .

[12]  J. Alderson.  Assessing Reading: Acknowledgements , 2000 .

[13]  Howard Wainer,et al.  Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies , 2011 .

[14]  A Jackson Stenner,et al.  How accurate are lexile text measures? , 2006, Journal of applied measurement.

[15]  John B. Carroll,et al.  The American Heritage Word Frequency Book , 1971 .

[16]  Richard Tretiak,et al.  Sentence Depth Measures as Predictors of Reading Difficulty. , 1971 .

[17]  D. Biber.  Spoken and Written Textual Dimensions in English: Resolving the Contradictory Findings , 1986 .

[18]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[19]  Michael Halliday,et al.  Cohesion in English , 1976 .

[20]  Maxine Eskénazi,et al.  Classroom success of an intelligent tutoring system for lexical practice and reading comprehension , 2006, INTERSPEECH.

[21]  J. Tukey.  The Future of Data Analysis , 1962 .

[22]  Ani Nenkova,et al.  Revisiting Readability: A Unified Framework for Predicting Text Quality , 2008, EMNLP.

[23]  Elfrieda H. Hiebert,et al.  Beyond Single Readability Measures: Using Multiple Sources of Information in Establishing Text Complexity , 2011 .

[24]  Elfrieda H. Hiebert,et al.  Toward a Theoretical Model of Text Complexity for the Early Grades: Learning From the Past, Anticipating the Future , 2012 .

[25]  Michael Flor,et al.  Lexical Tightness and Text Complexity , 2013 .

[26]  Diana Adler,et al.  Using Multivariate Statistics , 2016 .

[27]  Kathleen M. Sheehan,et al.  Measuring Cohesion: An Approach That Accounts for Differences in the Degree of Integration Challenge Presented by Different Types of Sentences , 2013 .

[28]  Walt Detmar Meurers,et al.  On Improving the Accuracy of Readability Classification using Insights from Second Language Acquisition , 2012, BEA@NAACL-HLT.

[29]  Victor H. Yngve,et al.  A model and an hypothesis for language structure , 1960 .

[30]  Pearson ’ s Text Complexity Measure , 2022 .

[31]  Averil Coxhead.  A New Academic Word List , 2000 .

[32]  Kevyn Collins-Thompson,et al.  Information retrieval for language tutoring: an overview of the REAP project , 2004, SIGIR '04.

[33]  Elfrieda H. Hiebert,et al.  Using Multiple Sources of Information in Establishing Text Complexity. Reading Research Report. #11.03. , 2011 .

[34]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[35]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[36]  Morton Ann Gernsbacher,et al.  Language Comprehension As Structure Building , 1990 .

[37]  Catherine Snow,et al.  Reading for Understanding: Toward an R&D Program in Reading Comprehension , 2002 .

[38]  Arthur C. Graesser,et al.  Coh-Metrix: Capturing Linguistic Features of Cohesion , 2010 .

[39]  Irene Kostin,et al.  Reading Level Assessment for Literary and Expository Texts , 2007 .

[40]  R. Viertl.  On the Future of Data Analysis , 2002 .

[41]  John Sabatini,et al.  Measuring up: Advances in How We Assess Reading Ability. , 2012 .

[42]  Michael Vitale,et al.  The Wisdom of Crowds , 2015, Cell.