How Does "Sentence Structure and Vocabulary" Function as a Scoring Criterion Alongside Other Criteria in Writing Assessment 1 ?

Several studies have evaluated sentence structure and vocabulary (SSV) as a scoring criterion in assessing writing, but no consensus on its functionality has been reached. The present study presents evidence that this scoring criterion may not be appropriate in writing assessment. Scripts by 182 ESL students at two language centers were analyzed with the Rasch partial credit model. Although other scoring criteria functioned satisfactorily, SSV scores did not fit the Rasch model, and analysis of residuals showed SSV scoring on most test prompts loaded on a benign secondary dimension. The study proposes that a lexico-grammatical scoring criterion has potentially conflicting properties, and therefore recommends considering separate vocabulary and grammar criteria in writing assessment.

[1]  A. Mehdi Riazi,et al.  Teacher- and peer-scaffolding behaviors : effects on EFL students' writing improvement , 2011 .

[2]  Alastair Pollitt,et al.  Calibrating graded assessments: Rasch partial credit analysis of performance in writing , 1987 .

[3]  Youn-Hee Kim,et al.  Diagnosing EAP writing ability using the Reduced Reparameterized Unified Model , 2011 .

[4]  Karen Draney,et al.  Objective measurement : theory into practice , 1992 .

[5]  Khalil Motallebzadeh,et al.  Models of Language Proficiency: a Reflection on the Construct of Language Ability , 2011 .

[6]  Kimi Kondo-Brown,et al.  A FACETS analysis of rater bias in measuring Japanese second language writing performance , 2002 .

[7]  Ute Knoch,et al.  Diagnostic assessment of writing: A comparison of two rating scales , 2009 .

[8]  Vahid Aryadoust.  Differential Item Functioning in While-Listening Performance Tests: The Case of the International English Language Testing System (IELTS) Listening Module , 2012 .

[9]  Peter Mickan,et al.  Study of response validity of the IELTS writing subtest , 2000 .

[10]  Kathleen Blake Yancey,et al.  Construct and Consequence: Validity in Writing Assessment , 2007 .

[11]  Richard M. Smith.  The Distributional Properties of Rasch Item Fit Statistics , 1991 .

[12]  Kristen di Gennaro.  Investigating differences in the writing performance of international and Generation 1.5 students , 2009 .

[13]  R. M. Smith,et al.  Fit analysis in latent trait measurement models. , 2000, Journal of applied measurement.

[14]  William Grabe,et al.  Communicative language proficiency : definition and implications for TOEFL 2000 , 1997 .

[15]  M. Scardamalia,et al.  The psychology of written composition , 1987 .

[16]  Andy Field,et al.  Discovering statistics using SPSS, 2nd ed. , 2005 .

[17]  Hayes.  identifying the organization of wi iiing processes , 1980 .

[18]  Liz Hamp-Lyons.  Assessing Second Language Writing in Academic Contexts , 1991 .

[19]  Ute Knoch,et al.  Diagnostic Writing Assessment: The Development and Validation of a Rating Scale , 2009 .

[20]  J. Hayes.  A new framework for understanding cognition and affect in writing. , 1996 .

[21]  T. Homburg.  Holistic Evaluation of ESL Compositions: Can It Be Validated Objectively? , 1984 .

[22]  Stuart D. Shaw,et al.  Examining Writing: Research and Practice in Assessing Second Language Writing , 2007 .

[23]  Everett V. Smith.  Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. , 2002, Journal of applied measurement.

[24]  Everett V. Smith,et al.  Introduction to Rasch measurement : theory, models and applications , 2004 .

[25]  George Engelhard,et al.  Evaluating Rater Accuracy in Performance Assessments. , 1996 .

[26]  U. Knoch,et al.  Validity and fairness implications of varying time conditions on a diagnostic test of academic English writing proficiency , 2010 .

[27]  George Engelhard,et al.  Examining Rater Errors in the Assessment of Written Composition With a Many-Faceted Rasch Model , 1994 .

[28]  W. Grabe,et al.  Theory and Practice of Writing: An Applied Linguistic Perspective , 1998 .

[29]  Peter Mickan.  'What's your score?': An investigation into language descriptors for rating written performance , 2003 .

[30]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[31]  Gillian Wigglesworth,et al.  Task.design in IELTS academic writing task 1: The effect of quantity and manner of presentation of information on candidate writing , 2003 .

[32]  John Hattie,et al.  Methodology Review: Assessing Unidimensionality of Tests and ltenls , 1985 .

[33]  Vanessa Jakeman,et al.  Cambridge practice tests for IELTS 1 , 1996 .

[34]  Kyle Perkins,et al.  Research in Language Testing , 1980 .

[35]  Charles Bazerman,et al.  Handbook of research on writing : history, society, school, individual, text , 2007 .

[36]  C. Fox,et al.  Applying the Rasch Model: Fundamental Measurement in the Human Sciences , 2001 .

[37]  Hidetoshi Saito,et al.  EFL classroom peer assessment: Training effects on rating and commenting , 2008 .

[38]  Sara Cushing Weigle,et al.  Assessing Writing: Series Editors' Preface , 2002 .

[39]  C. Michael Levy,et al.  The Science of Writing : Theories, Methods, Individual Differences and Applications , 1996 .

[40]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[41]  E. Schaefer.  Rater bias patterns in an EFL writing assessment , 2008 .

[42]  Thomas Eckes,et al.  Examining Rater Effects in TestDaF Writing and Speaking Performance Assessments: A Many-Facet Rasch Analysis , 2005 .

[43]  Lyle F. Bachman.  语言测试要略 = Fundamental considerations in language testing , 1990 .

[44]  Sara Cushing Weigle,et al.  Using FACETS to model rater training effects , 1998 .

[45]  M. Fahim,et al.  The Effects of Rater Training on Raters' Severity and Bias in Second Language Writing Assessment , 2011 .

[46]  T. McNamara.  Item Response Theory and the validation of an ESP test for health professionals , 1990 .

[47]  Annie Brown,et al.  Candidate discourse in the revised IELTS Speaking Test , 2006 .

[48]  Peter Mickan,et al.  Text analysis and the assessment of academic writing , 2003 .

[49]  T. McNamara.  Measuring Second Language Performance , 1996 .

[50]  G. Heath.  Writing , 1971, Veterinary Record.

[51]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[52]  Brian K. Lynch,et al.  Using G-theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants , 1998 .

[53]  Anne Margaret Smith,et al.  Documenting features of written language production typical at different IELTS band score levels , 2007 .

[54]  Lin Lougheed.  Barron's How to Prepare for the Computer-Based Toefl Essay: Test of English As a Foreign Language , 2000 .

[55]  George Engelhard,et al.  The Measurement of Writing Ability With a Many-Faceted Rasch Model , 1992 .

[56]  Ken Hyland,et al.  Teaching and Researching Writing , 2001 .

[57]  Vahid Aryadoust,et al.  An Investigation of Differential Item Functioning in the MELAB Listening Test , 2011 .

[58]  A Binomial Trials Model for Examining the Ratings of Standard-Setting Judges. , 1998 .

[59]  Paul Nation,et al.  An investigation of the lexical dimension of the IELTS Speaking Test , 2006 .