Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom

The Writing Pal (W-Pal) is a novel intelligent tutoring system (ITS) that offers writing strategy instruction, game-based practice, essay writing practice, and formative feedback to developing writers. Compared to more tractable and constrained learning domains for ITS, writing is an ill-defined domain because the features of effective writing are difficult to quantify and individual writers can employ diverse strategies to achieve similar goals. The development of an ITS in an ill-defined domain presents particular challenges regarding comprehensive instruction, modularized content, extended practice, and formative feedback. In this article, we describe how the development of W-Pal has uniquely addressed these concerns and present the results of a study assessing the feasibility of this system in high school English classrooms. This study included 2 teachers and their 141 10th grade English class students who utilized W-Pal over a 6-month period during the academic year. Log-file analyses showed that students used all aspects of W-Pal, but activity and engagement was uneven throughout the year and decreased over time. Essay scores improved over time and surveys indicated that students perceived the lessons, games, and feedback as beneficial. However, specific aspects of the learning environment were critiqued as annoying, challenging, or lacking specificity. Overall, the results suggest that the system was generally well-received by the students but also offer insights for the development of ITSs in ill-defined domains.

[1]  Danielle S. McNamara,et al.  Improving Adolescent Students' Reading Comprehension with Istart , 2006 .

[2]  Philip M. McCarthy,et al.  Linguistic Features of Writing Quality , 2010 .

[3]  Scotty D. Craig,et al.  Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , 2004 .

[4]  Kurt VanLehn,et al.  The Design of Self-explanation Prompts: The Fit Hypothesis , 2009 .

[5]  Danielle S. McNamara,et al.  Computer-based scaffolding to facilitate students' development of expertise in academic writing , 2012 .

[6]  Thomas Quinlan,et al.  Does Automated Feedback Help Students Learn to Write? , 2010 .

[7]  Frank Pajares,et al.  SELF-EFFICACY BELIEFS, MOTIVATION, AND ACHIEVEMENT IN WRITING: A REVIEW OF THE LITERATURE , 2003 .

[8]  Michelle Meadows,et al.  A review of the literature on marking reliability , 2005 .

[9]  Sharlene A. Kiuhara,et al.  A Meta-Analysis of Writing Instruction for Students in the Elementary Grades. , 2012 .

[10]  Martin Chodorow,et al.  Automated Essay Evaluation: The Criterion Online Writing Service , 2004, AI Mag..

[11]  Beverly Park Woolf,et al.  Evaluation of AnimalWatch: An Intelligent Tutoring System for Arithmetic and Fractions , 2010 .

[12]  Danielle S. McNamara,et al.  The Development of Writing Proficiency as a Function of Grade Level: A Linguistic Analysis , 2011 .

[13]  Vincent Aleven,et al.  An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor , 2002, Cogn. Sci..

[14]  Danielle S. McNamara,et al.  Internal Usability Testing of Automated Essay Feedback in an Intelligent Writing Tutor , 2011, FLAIRS Conference.

[15]  Heather H. Mitchell,et al.  AutoTutor: A tutor with dialogue in natural language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[16]  Danielle S. McNamara,et al.  Predicting Human Scores of Essay Quality Using Computational Indices of Linguistic and Textual Features , 2011, AIED.

[17]  K. VanLehn,et al.  Scaffolding Deep Comprehension Strategies Through Point&Query, AutoTutor, and iSTART , 2005 .

[18]  Mark Warschauer,et al.  Automated writing evaluation: defining the classroom research agenda , 2006 .

[19]  Paul Deane,et al.  COGNITIVE MODELS OF WRITING: WRITING PROFICIENCY AS A COMPLEX INTEGRATED SKILL , 2008 .

[20]  Arthur C. Graesser,et al.  The Writing-Pal: Natural Language Algorithms to Support Intelligent Tutoring on Writing Strategies , 2012 .

[21]  W. Lewis Johnson,et al.  Assessing Aptitude for Learning with a Serious Game for Foreign Language and Culture , 2008, Intelligent Tutoring Systems.

[22]  J. Hayes,et al.  A Cognitive Process Theory of Writing , 1981, College Composition & Communication.

[23]  Vincent Aleven,et al.  Concepts, Structures, and Goals: Redefining Ill-Definedness , 2009, Int. J. Artif. Intell. Educ..

[24]  V. Shute Focus on Formative Feedback , 2008 .

[25]  David Malvern,et al.  Lexical Diversity and Language Development: Quantification and Assessment , 2004 .

[26]  Danielle S McNamara,et al.  Natural language processing in an intelligent writing strategy tutoring system , 2012, Behavior Research Methods.

[27]  Laura K. Allen,et al.  The Writing Pal Intelligent Tutoring System: Usability Testing and Development , 2014 .

[28]  T. Landauer Automatic Essay Assessment , 2003 .

[29]  Michael Glass,et al.  Learning from a Computer Tutor with Natural Language Capabilities , 2003, Interact. Learn. Environ..

[30]  B. Huot Toward a New Theory of Writing Assessment , 1996 .

[31]  Lawrence M. Rudner,et al.  An Evaluation of the IntelliMetric[SM] Essay Scoring System. , 2006 .

[32]  M. Warschauer,et al.  Automated Writing Assessment in the Classroom , 2008 .

[33]  Mark Warschauer,et al.  Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. , 2010 .

[34]  Kurt VanLehn,et al.  The Andes Physics Tutoring System: Lessons Learned , 2005, Int. J. Artif. Intell. Educ..

[35]  Herbert A. Simon,et al.  The Structure of Ill Structured Problems , 1973, Artif. Intell..

[36]  Kurt Rowley,et al.  The Effect of a Computer Tutor for Writers on Student Writing Achievement , 2003 .

[37]  George Hillocks,et al.  What Works in Teaching Composition: A Meta-Analysis of Experimental Treatment Studies , 1984, American Journal of Education.

[38]  Hedy McGarrell,et al.  Motivating revision of drafts through formative feedback , 2007 .

[39]  Danielle S. McNamara,et al.  Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing , 2011 .

[40]  S. Graham,et al.  A meta-analysis of writing instruction for adolescent students. , 2007 .

[41]  Stephen Graham,et al.  Explicitly Teaching Strategies, Skills, and Knowledge: Writing Instruction in Middle School Classrooms. , 2002 .

[42]  Summary Street®: Computer-guided Summary writing , 2007 .

[43]  Michael Albrecht,et al.  The efficacy of a Web-based counterargument tutor , 2009, Behavior research methods.

[44]  G. T. Jackson,et al.  Motivation and performance in a game-based intelligent tutoring system , 2013 .

[45]  Johann Gamper,et al.  A Review of Intelligent CALL Systems , 2002 .

[46]  Semire Dikli,et al.  An Overview of Automated Scoring of Essays. , 2006 .

[47]  D. McNamara,et al.  Integrating iSTART into a High School Curriculum , 2007 .

[48]  Karen E. Lochbaum,et al.  A New Formative Assessment Technology for Reading and Writing , 2009 .

[49]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[50]  Lawrence M. Rudner,et al.  An Evaluation of IntelliMetric™ Essay Scoring System , 2006 .