Associated effects of automated essay evaluation software on growth in writing quality for students with and without disabilities

The present study examined growth in writing quality associated with feedback provided by an automated essay evaluation system called PEG Writing. Equal numbers of students with disabilities (SWD) and typically-developing students (TD) matched on prior writing achievement were sampled (n = 1196 total). Data from a subsample of students (n = 655) was used to investigate evidence of transfer to improved first-draft performance on a follow up writing prompt. Three-level hierarchical linear modeling was used. Findings indicated that SWD produced first drafts of lesser quality than TD students, but grew at a faster rate and were able to close the gap in writing quality after five revisions. However, these effects were moderated by school quality and the availability of internet-connected devices in schools. There was no evidence of transfer for either group of students. Results document a positive association between the use of PEG Writing and growth in writing quality for SWD, and underscore the importance of having sufficient technology resources for maximizing this growth.

[1]  Sharlene A. Kiuhara,et al.  A Meta-Analysis of Writing Instruction for Students in the Elementary Grades. , 2012 .

[2]  Julie E. Dockrell,et al.  The Impact of Specific Language Impairment on Adolescents' Written Text , 2009 .

[3]  R. T. Kellogg,et al.  Training Advanced Writing Skills: The Case for Deliberate Practice , 2009 .

[4]  Karen R. Harris,et al.  Assessing the Writing Achievement ofYoung Struggling Writers: Application ofGeneralizability Theory , 2016 .

[5]  Frank Pajares,et al.  SELF-EFFICACY BELIEFS, MOTIVATION, AND ACHIEVEMENT IN WRITING: A REVIEW OF THE LITERATURE , 2003 .

[6]  Mark Warschauer,et al.  Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. , 2010 .

[7]  C. MacArthur.  Reflections on Research on Writing and Technology for Struggling Writers , 2009 .

[8]  Michael Hebert,et al.  Informing writing: The benefits of formative assessment. A Carnegie Corporation Time to Act report , 2011 .

[9]  E. B. Page.  Project Essay Grade: PEG. , 2003 .

[10]  Douglas Biber,et al.  THE EFFECTIVENESS OF FEEDBACK FOR L1‐ENGLISH AND L2‐WRITING DEVELOPMENT: A META‐ANALYSIS , 2011 .

[11]  Natalie G. Olinghouse,et al.  Teaching Elementary School Students to Be Effective Writers: A Practice Guide. NCEE 2012-4058. , 2012 .

[12]  S. Graham,et al.  Students with learning disabilities and the process of writing: A meta-analysis of SRSD studies. , 2003 .

[13]  Natalie G. Olinghouse,et al.  Fifth-Grade Students' Knowledge about Writing Process and Writing Genres , 2013, The Elementary School Journal.

[14]  John R. Hayes,et al.  What Triggers Revision , 2004 .

[15]  Charles A. MacArthur,et al.  Effects of goal setting and procedural facilitation on the revising behavior and writing performance of students with writing and learning problems. , 1995 .

[16]  Natalie G. Olinghouse,et al.  The relationship between vocabulary and writing quality in three genres , 2012, Reading and Writing.

[17]  Charles A. MacArthur,et al.  Knowledge of Revision and Revising Behavior among Students with Learning Disabilities , 1991 .

[18]  Mark D. Shermis,et al.  State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration , 2014 .

[19]  Gilbert N. Andrada,et al.  Using Automated Feedback to Improve Writing Quality: Opportunities and Challenges , 2016 .

[20]  S. Graham,et al.  Common Core State Standards, Writing, and Students with LD: Recommendations , 2013 .

[21]  Deborah McCutchen,et al.  From novice to expert: Implications of language skills and writing-relevant knowledge for memory during the development of writing skill , 2011 .

[22]  J. Miller,et al.  The Nation's Report Card[TM]: Writing 2007. National Assessment of Educational Progress at Grades 8 and 12. National, State, and Trial Urban District Results. NCES 2008-468. , 2008 .

[23]  Steve Graham,et al.  Teaching writing to middle school students: a national survey , 2013, Reading and Writing.

[24]  Russell Gersten,et al.  Teaching Expressive Writing to Students with Learning Disabilities: A Meta-Analysis , 2001, The Elementary School Journal.

[25]  M. Scardamalia,et al.  The psychology of written composition , 1987 .

[26]  Steve Graham,et al.  “An adjective is a word hanging down from a noun”: Learning to write and students with learning disabilities , 2011, Annals of Dyslexia.

[27]  S. Graham,et al.  A Meta-Analysis of Writing Interventions for Students With Learning Disabilities , 2014 .

[28]  Sharon Vaughn,et al.  The Underlying Message in LD Intervention Research: Findings from Research Syntheses , 2000 .

[29]  Jill Burstein,et al.  Automated Essay Scoring : A Cross-disciplinary Perspective , 2003 .

[30]  Gary A. Troia,et al.  Development of Writing Knowledge in Grades 2–8: A Comparison of Typically Developing Writers and Their Struggling Peers , 2007 .

[31]  Carol Sue Englert,et al.  The Learning-to-Learn Strategies of Adolescent Students With Disabilities , 2009 .

[32]  Virginia W. Berninger,et al.  Computer instruction in handwriting, spelling, and composing for students with specific learning disabilities in grades 4-9 , 2015, Comput. Educ..

[33]  Virginia W. Berninger,et al.  Longitudinal Relationships of Levels of Language in Writing and between Writing and Reading in Grades 1 to 7. , 2010 .

[34]  Steve Graham,et al.  Writing Next: Effective Strategies to Improve Writing of Adolescents in Middle and High Schools. A Report to Carnegie Corporation of New York. , 2007 .

[35]  Steve Graham,et al.  Formative Assessment and Writing , 2015, The Elementary School Journal.

[36]  Charles A. MacArthur,et al.  Insights from Instructional Research on Revision with Struggling Writers , 2004 .

[37]  Joshua Wilson,et al.  Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality , 2016, Comput. Educ..

[38]  J. Hayes.  Modeling and Remodeling Writing , 2012 .

[39]  Virginia W. Berninger,et al.  Development of Language by Hand and Its Connections with Language by Ear, Mouth, and Eye , 2000 .

[40]  Gary A. Troia,et al.  Instruction and Assessment for Struggling Writers Evidence-Based Practices. , 2009 .

[41]  David Vaughn,et al.  Automated Scoring in Assessment Systems , 2016 .

[42]  Jill Burstein,et al.  Handbook of Automated Essay Evaluation Current Applications and New Directions , 2018 .

[43]  Thomas Quinlan,et al.  Does Automated Feedback Help Students Learn to Write? , 2010 .

[44]  S. Graham,et al.  Teaching Writing to Elementary Students in Grades 4–6: A National Survey , 2010, The Elementary School Journal.

[45]  Stephen Graham,et al.  Word processing programs and weaker writers/readers: a meta-analysis of research findings , 2012 .

[46]  William Wresch,et al.  The Imminence of Grading Essays by Computer-25 Years Later , 1993 .

[47]  Lindsay Clare Matsumura,et al.  Teacher Feedback, Writing Assignment Quality, and Third-Grade Students' Revision in Lower-And Higher-Achieving Urban Schools , 2002, The Elementary School Journal.

[48]  Joshua Wilson,et al.  Does Automated Feedback Improve Writing Quality , 2014 .

[49]  Ben Hamner,et al.  Contrasting state-of-the-art automated scoring of essays: analysis , 2012 .

[50]  L. Paganucci,et al.  Do Digital Writing Tools Deliver? Student Perceptions of Writing Quality Using Digital Tools and Online Writing Environments , 2015 .

[51]  M. Warschauer,et al.  Automated Writing Assessment in the Classroom , 2008 .

[52]  Scott Dooley,et al.  Summary Street®: Computer Support for Comprehension and Writing , 2005 .

[53]  T. Keith,et al.  Trait Ratings for Automated Essay Grading , 2002 .

[54]  Sandra Murphy,et al.  Designing Writing Tasks for the Assessment of Writing , 1988 .

[55]  J. Singer,et al.  Applied Longitudinal Data Analysis , 2003 .

[56]  P. Alexander.  The Development of Expertise: The Journey From Acclimation to Proficiency , 2003 .

[57]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[58]  Aek Phakiti,et al.  The effects of computer-generated feedback on the quality of writing , 2014 .

[59]  Judith A. Langer,et al.  What Is Happening in the Teaching of Writing , 2009 .

[60]  N. S Moore,et al.  Student Use of Automated Essay Evaluation Technology during Revision , 2016 .