The influence of lexical features on teacher judgements of ESL argumentative essays

Abstract Numerous studies have examined the relationship between lexical features of students’ compositions and judgements of text quality. However, the degree to which teachers’ judgements are influenced by the quality of vocabulary in students’ essays with regard to their assessment of other textual characteristics is relatively unexplored. This experimental study investigates the influence of lexical features on teachers’ judgements of English as a second language (ESL) argumentative essays. Using analytic and holistic rating scales, English pre-service teachers (N = 37) in Switzerland assessed four essays of different proficiency levels in which the levels of lexical diversity and sophistication had been experimentally varied. Coh-Metrix software was used to manipulate the level of lexical diversity, as measured by MTLD and D, and the Tool for the Automatic Analysis of Lexical Sophistication (TAALES) software was used to obtain differing levels of lexical sophistication, as measured by word range. The results suggested that texts with greater lexical diversity and sophistication were assessed more positively concerning their overall quality as well as the analytic criteria ‘grammar’ and ‘frame of essay’. The implications of this study for classroom practice and teacher education are discussed.

[1]  J. Baumert,et al.  Stichwort: Professionelle Kompetenz von Lehrkräften , 2006 .

[2]  Moira Linnarud.  Lexis in composition : a performance analysis of Swedish learners' written English , 1986 .

[3]  A. Bussu.  Introduction , 2018, Police Practice and Research.

[4]  Thomas Eckes,et al.  Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater-Mediated Assessments , 2011 .

[5]  Scott A. Crossley,et al.  Automatically Assessing Lexical Sophistication: Indices, Tools, Findings, and Application , 2015 .

[6]  Helen Timperley,et al.  Feedback to Writing, Assessment for Teaching and Learning and Student Progress. , 2010 .

[7]  M. Scardamalia,et al.  The psychology of written composition , 1987 .

[8]  D. Sadler.  Formative assessment and the design of instructional systems , 1989 .

[9]  Danielle S. McNamara,et al.  Predicting the proficiency level of language learners using lexical indices , 2012 .

[10]  Alister Cumming,et al.  Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL , 2005 .

[11]  Alister Cumming,et al.  Decision Making While Rating ESL/EFL Writing Tasks: A Descriptive Framework. , 2002 .

[12]  R. Hawkey,et al.  Developing a common scale for the assessment of writing , 2004 .

[13]  M. Meadows,et al.  The effect of marker background and training on the quality of marking in GCSE English , 2010 .

[14]  Danielle S. McNamara,et al.  Chapter 4. Validating lexical measures using human scores of lexical proficiency , 2013 .

[15]  Scott Jarvis,et al.  Short texts, best-fitting curves and new measures of lexical diversity , 2002 .

[16]  Hong Jiao,et al.  Features of difficult-to-score essays , 2016 .

[17]  Helmut Daller,et al.  Modelling and Assessing Vocabulary Knowledge: Fundamental issues , 2007 .

[18]  Friedrich-Wilhelm Schrader.  Diagnostische Kompetenz von Lehrpersonen , 2013, BzL - Beiträge zur Lehrerinnen- und Lehrerbildung.

[19]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[20]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[21]  J. Hayes,et al.  A Cognitive Process Theory of Writing , 1981, College Composition & Communication.

[22]  P. Nation,et al.  Vocabulary size and use: Lexical richness in L2 written production , 1995 .

[23]  Sara Cushing Weigle,et al.  Teaching writing teachers about assessment , 2007 .

[24]  U. Knoch.  Assessing writing , 2021, The Routledge Handbook of Language Testing.

[25]  N. Verhelst,et al.  Common European Framework of Reference for Languages: learning, teaching, assessment , 2009 .

[26]  E. Brunswik.  Perception and the Representative Design of Psychological Experiments , 1957 .

[27]  Ali Reza Rezaei,et al.  Reliability and validity of rubrics for assessment through writing , 2010 .

[28]  Danielle S. McNamara,et al.  Assessing Lexical Proficiency Using Analytic Ratings: A Case for Collocation Accuracy , 2014 .

[29]  Philip M. McCarthy,et al.  Linguistic Features of Writing Quality , 2010 .

[30]  Stefanie A. Wind,et al.  Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments , 2017 .

[31]  Sue Bennett,et al.  Support for assessment practice: developing the Assessment Design Decisions Framework , 2016 .

[32]  Guoxing Yu,et al.  Lexical Diversity in Writing and Speaking Task Performances , 2010 .

[33]  Yogendra Patil,et al.  Exploring the relationship between textual characteristics and rating quality in rater-mediated writing assessments: An illustration with L1 and L2 writing assessments , 2017 .

[34]  Jeanine Treffers-Daller.  Measuring lexical diversity among L2 learners of French: an exploration of the validity of D, MTLD and HD-D as measures of language ability , 2013 .

[35]  J. Möller,et al.  Das Schülerinventar: Welche Schülermerkmale die Leistungsurteile von Lehrkräften beeinflussen , 2015 .

[36]  Arthur C. Graesser,et al.  Automated Evaluation of Text and Discourse with Coh-Metrix: List of Tables , 2014 .

[37]  Xiaofei Lu.  The Relationship of Lexical Richness to the Quality of ESL Learners' Oral Narratives. , 2012 .

[38]  Scott Jarvis,et al.  Exploring multiple profiles of highly rated learner compositions , 2003 .

[39]  D. Ferris,et al.  Teacher commentary on student writing: Descriptions & implications , 1997 .

[40]  Danielle S. McNamara,et al.  Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing , 2011 .

[41]  Laura K. Allen,et al.  Linguistic Microfeatures to Predict L2 Writing Proficiency: A Case Study in Automated Writing Evaluation. , 2014 .

[42]  Stuart Webb,et al.  Researching and Analyzing Vocabulary , 2010 .

[43]  L. Hamp-Lyons.  Exploring the Dynamics of Second Language Writing: Writing teachers as assessors of writing , 2003 .

[44]  Emanuel Schmider,et al.  Is It Really Robust , 2010 .

[45]  Jens Möller,et al.  Accuracy of teachers' judgments of students' academic achievement: A meta-analysis , 2012 .

[46]  P. Black,et al.  'In praise of educational research': formative assessment , 2003 .

[47]  Leslie Grant,et al.  Using Computer-Tagged Linguistic Features to Describe L2 Writing Differences , 2000 .

[48]  M. Coe,et al.  An Investigation of the Impact of the 6+1 Trait Writing Model on Grade 5 Student Writing Achievement. Final Report. NCEE 2012-4010. , 2011 .

[49]  Michael Lewis.  The Lexical Approach: The State of ELT and a Way Forward , 2002 .

[50]  Tzipora Rakedzon,et al.  To make a long story short: A rubric for assessing graduate students’ academic and popular science writing skills , 2017 .

[51]  Deborah J. Crusan,et al.  Writing Assessment Literacy: Surveying Second Language Teachers' Knowledge, Beliefs, and Practices , 2016 .

[52]  M. Scriven.  The methodology of evaluation , 1966 .

[53]  Dale P. Scannell,et al.  The Effect of Selected Composition Errors on Grades Assigned to Essay Examinations , 1966 .

[54]  Randi Reppen,et al.  Understanding first-year L2 writing: A lexico-grammatical analysis across L1s, genres, and language ratings , 2016 .

[55]  Composition Errors and Essay Examination Grades Re-Examined , 1967 .

[56]  Danielle S. McNamara,et al.  Predicting Second Language Writing Proficiency: The Roles of Cohesion and Linguistic Sophistication , 2012 .

[57]  Scott A. Crossley,et al.  The relationship between lexical sophistication and independent and source-based writing , 2016 .

[58]  J. Möller,et al.  The Effects of Student Characteristics on Teachers’ Judgment Accuracy: Disentangling Ethnicity, Minority Status, and Achievement , 2017 .

[59]  Scott Jarvis,et al.  vocd: A theoretical and empirical evaluation , 2007 .

[60]  T. Bechger,et al.  Detecting Halo Effects in Performance-Based Examinations , 2010 .

[61]  Natalie G. Olinghouse,et al.  The relationship between vocabulary and writing quality in three genres , 2012, Reading and Writing.

[62]  E. Thorndike.  A constant error in psychological ratings. , 1920 .

[63]  Ken Hyland,et al.  Second Language Writing , 2003 .

[64]  Cheryl A. Engber.  The relationship of lexical proficiency to the quality of ESL compositions , 1995 .

[65]  Deborah J. Crusan.  Assessment in the Second Language Writing Classroom , 2010 .

[66]  Ute Knoch,et al.  Diagnostic assessment of writing: A comparison of two rating scales , 2009 .

[67]  David Malvern,et al.  Developmental trends in lexical diversity , 2004 .

[68]  Elana Shohamy,et al.  The Effect of Raters' Background and Training on the Reliability of Direct Writing Tests , 1992 .

[69]  J. Alderson,et al.  Towards a Theory of Diagnosis in Second and Foreign Language Assessment: Insights from Professional Practice Across Diverse Fields , 2015 .

[70]  Danielle S. McNamara,et al.  Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study , 2013 .

[71]  Arthur C. Graesser,et al.  Coh-Metrix: Analysis of text on cohesion and language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[72]  Wei-Na Zhu.  Performing Argumentative Writing in English: Difficulties, Processes, and Strategies , 2001 .

[73]  Philip M. McCarthy,et al.  MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment , 2010, Behavior research methods.

[74]  Danielle S. McNamara,et al.  Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications , 2013 .

[75]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[76]  David Malvern,et al.  Lexical Diversity and Language Development: Quantification and Assessment , 2004 .

[77]  L. Shulman.  Knowledge and Teaching: Foundations of the New Reform , 1987 .

[78]  Is Teaching Experience Necessary for Reliable Scoring of Extended English Questions , 2009 .

[79]  Karyn N. Erkfritz-Gay,et al.  Purposes of Assessment , 2016 .

[80]  Thomas Eckes,et al.  Rater types in writing performance assessments: A classification approach to rater variability , 2008 .

[81]  K. Hyland,et al.  A Genre Description of the Argumentative Essay , 1990 .