Using Automated Indices of Cohesion to Evaluate an Intelligent Tutoring System and an Automated Writing Evaluation System

We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.

[1]  Janet Lesley Giltrow.  Academic writing : writing and reading in the disciplines , 2002 .

[2]  Arthur C. Graesser,et al.  Coh-Metrix: Analysis of text on cohesion and language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[3]  G. Kootstra,et al.  Proceedings of the 28th Annual Conference of the Cognitive Science Society , 2006 .

[4]  Mark Warschauer,et al.  Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. , 2010 .

[5]  S. Graham,et al.  A meta-analysis of writing instruction for adolescent students. , 2007 .

[6]  Danielle S McNamara,et al.  Natural language processing in an intelligent writing strategy tutoring system , 2012, Behavior Research Methods.

[7]  Danielle S. McNamara,et al.  Developing Pedagogically-Guided Threshold Algorithms for Intelligent Automated Essay Feedback , 2012, FLAIRS Conference.

[8]  D. McNamara,et al.  Cohesion, coherence, and expert evaluations of writing proficiency , 2010 .

[9]  Danielle S. McNamara,et al.  The Development of Writing Proficiency as a Function of Grade Level: A Linguistic Analysis , 2011 .

[10]  Terry D. Warfield,et al.  Effects of Repeated Practice and Contextual-Writing Experiences on College Students' Writing Skills. , 2002 .

[11]  S. Graham,et al.  The Role of Self-Regulation and Transcription Skills in Writing and Writing Development , 2000 .

[12]  Arthur C. Graesser,et al.  Using LSA to Automatically Identify Givenness and Newness of Noun Phrases in Written Discourse , 2005 .

[13]  W. Kintsch,et al.  Are Good Texts Always Better? Interactions of Text Coherence, Background Knowledge, and Levels of Understanding in Learning From Text , 1996 .

[14]  Danielle S. McNamara,et al.  Text Coherence and Judgments of Essay Quality: Models of Quality and Coherence , 2011, CogSci.

[15]  Hedy McGarrell,et al.  Motivating revision of drafts through formative feedback , 2007 .

[16]  R. T. Kellogg,et al.  Improving the writing skills of college students , 2007, Psychonomic bulletin & review.

[17]  Arthur C. Graesser,et al.  The Writing-Pal: Natural Language Algorithms to Support Intelligent Tutoring on Writing Strategies , 2012 .

[18]  Danielle S. McNamara,et al.  Predicting Second Language Writing Proficiency: The Roles of Cohesion and Linguistic Sophistication , 2012 .