Towards Effective Integration and Positive Impact of Automated Writing Evaluation in L2 Writing

The increasing dominance of English has elevated the need to develop an ability to effectively communicate in writing, and this has put a strain on second language education programs worldwide. Faced with time-consuming and copious commenting on student drafts and inspired by the promise of computerized writing assessment, many "educational technology enthusiasts are looking to A WE [automated writing evaluation] as a silver bullet for language and literacy development" (Warschauer & Ware, 2006, p. 175). This chapter reviews what AWE offers for learners and teachers and raises a number of controversies regarding A WE effectiveness with the underlying message that clear milestone targets need to be set with respect to A WE development, implementation, and evaluation in order to ensure positive impact of this technology on L2 writing. In support of this message, the chapter introduces an examplelADE, a prototype of contextbased A WE conceptualized and operationalized to address latent issues through a synthesis of theoretical premises and learning needs. Multifaceted empirical evaluation of lADE further provides insights into processes triggered by interaction with A WE technology and foregrounds a call for future research needed to inform effective application of AWE in L2 writing classrooms. 1. Automated Writing Evaluation for Learning and Teaching Automated writing evaluation (AWE), defined as "the ability of computer technology to evaluate and score written prose" (Shermis & Burstein, 2003, p. xiii), is informed by educational measurement, computational linguistics as well as cognitive science and pedagogy. In other words, psychometric evaluations of reliability and validity, considerations about intelligent operational systems and their functionality as well as models that reflect thought processes and factors considered to be most beneficial for learners have all contributed to the development of AWE systems. 82 AUTOMATED WRITING EVALUATION This technology originated from automated scoring engines and was initially referred to as computerized essay scoring, computer essay grading, computer-assisted writing assessment, or machine scoring of essays. The first scoring system, Project Essay Gride (PEG), was developed in the 1960s and employed multiple regression analysis to predict scores based on measurable variables in the form of surface linguistic features (Page, 1994). The systems following PEG-Intelligent Essay Assessor (lEA), Electronic Essay Rater (e-rater), Conceptual Rater (c-rater), Schema Extract Analyze and Report (SEAR), Paperless School freetext Marking Engine (PS-ME), Automark, and AntMover-assess written constructed responses using natural language processing (NLpl) in combination with statistical techniques that analyze a wide range of aspects of writing constructs such as grammar, syntactic complexity, mechanics, style, topical content, content development, deviance, and so on. Before long, the systems were reconfigured to generate intelligent feedback on all these features. These systems led to the development of A WE products like the pioneering Writer's Workbench (MacDonald, Frase, Gingrich, & Keenan, 1982) and leadingedge programs like Criterion, MY Access! (for further information on this writing program see Ware & Rivas's chapter in this volume), WriteToLearn, Summary Street, and Holt Online Essay Assessor.3 These commercially available products are being increasingly used in writing classrooms, shifting the role of AWE from pure assessment to assistance for learning (Chen & Cheng, 2008; Warschauer & Grimes, 2008) by offering both automated feedback and a wide range of complementary tools and features intended to help students (Burstein, Chodorow & Leacock, 2004; Pearson Education, 2007; Vantage Learning, 2007). In Criterion, for example, the students can solicit and receive feedback from their teacher through the program's interface. This helps them to focus not only on automatically detectable errors, but also on other, more subtle, aspects of writing identified by the teacher. Students can also view their performance summary, which includes a holistic score, the number of errors, and links to detailed feedback on each error category. In addition, Criterion has a context-sensitive Writer's Handbook that provides additional definitions and lessons. To assist students in their planning process, this program also offers a •Make a plan' tool with a choice of eight templates for planning strategies. MY Access!, in turn, has an online writing coach which evaluates student writing and provides revision goals and remediation activities for each of the writing traits, as well as an editor which highlights errors and provides editing suggestions. It also offers a writer's checklist for guidance, scoring rubrics for self-assessment, word banks for appropriate vocabulary use, and graphical pre writing tools for better formulation and organization of ideas. WriteToLearn has similar options; plus, it allows students to hear the text in read1 See Landauer, Laham, and Foltz (2003), Attali and Burstein (2006), Burstein, Leacock, and Swartz (2001), Christie (1999), Mason and Grove-Stephenson (2002), Mitchell, Russell, Broornhead, and Aldridge (2002), and Anthony (2003), respectively. 2 NLP is a branch of Artificial Intelligence. 3 For comprehensive reviews of these automated scoring systems see Chapelle and Chung (2010), Dildi (2006), Phillips (2007), and Valenti, Neri, and Cucchiarelli (2003).