New Spaces and Old Places: An Analysis of Writing Assessment Software

Abstract This article examines the strengths and weaknesses of emerging writing assessment technologies. Instead of providing a comprehensive review of each program, we take a deliberately selective approach using three key understandings about writing as a framework for analysis: writing is a socially situated activity; writing is functionally and formally diverse; and writing is a meaning-making activity that can be conveyed in multiple modalities. We conclude that the programs available today largely neglect the potential of emerging technologies to promote a broader vision of writing. Instead, they tend to align with the narrow view of writing dominant in a more recent era of testing and accountability, a view that is increasingly thrown into question. New technologies, we conclude, are for the most part being used to reinforce old practices. At a time when computer technology is increasingly looked to as a way to improve assessment, these findings have important implications.

[1]  Pamela J. Hinds,et al.  Workplace Studies: Recovering Work Practice and Informing System Design, Paul Luff, Jon Hindmarsh and Christian C. Heath (eds.) , 2000, Computer Supported Cooperative Work (CSCW).

[2]  S. Michaels,et al.  A pedagogy of Multiliteracies Designing Social Futures , 1996 .

[3]  Kathleen Blake Yancey,et al.  Writing in the 21st Century. , 2009 .

[4]  K. LeFevre,et al.  Invention as a Social Act , 1986 .

[5]  Jinhao Wang,et al.  Automated Essay Scoring Versus Human Scoring: A Comparative Study , 2007 .

[6]  Edward Brent,et al.  Time-Shifted Online Collaboration: Creating Teachable Moments Through Automated Grading , 2010 .

[7]  Cynthia L. Selfe,et al.  CCCC Position Statement on Teaching, Learning, and Assessing Writing in Digital Environments , 2004, College Composition & Communication.

[8]  Anne Beaufort,et al.  Writing in the Real World: Making the Transition from School to Work , 1999 .

[9]  Dixie Goswami,et al.  Writing in Non-Academic Settings. , 1981 .

[10]  B. Cope,et al.  Multiliteracies: Literacy Learning and the Design of Social Futures , 2001 .

[11]  Jill Burstein,et al.  The E-rater® scoring engine: Automated essay scoring with natural language processing. , 2003 .

[12]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[13]  Jay L. Lemke,et al.  Travels in hypermodality , 2002 .

[14]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[15]  K. Yancey Made Not Only in Words: Composition in a New Key , 2004 .

[16]  Jody Shipka Negotiating Rhetorical, Material, Methodological, and Technological Difference: Evaluating Multimodal Designs. , 2009 .

[17]  Charles A. MacArthur,et al.  Handbook of Writing Research , 2005 .

[18]  Angel A. Juan Monitoring and Assessment in Online Collaborative Environments: Emergent Computational Technologies for E-learning Support , 2009 .

[19]  George Hillocks,et al.  The Testing Trap: How State Writing Assessments Control Learning , 2002 .

[20]  Paul A Prior,et al.  From Speech Genres to Mediated Multimodal Genre Systems: Bakhtin, Voloshinov, and the Question of Writing , 2009 .

[21]  G. Kress Multimodality: A Social Semiotic Approach to Contemporary Communication , 2009 .

[22]  Deborah Brandt,et al.  Literacy as Involvement: The Acts of Writers, Readers, and Texts , 1990 .

[23]  B. Street Literacy in Theory and Practice , 1984 .

[24]  B. Cope,et al.  “Multiliteracies”: New Literacies, New Learning , 2009 .

[25]  Steve Graham,et al.  Writing Next: Effective Strategies to Improve Writing of Adolescents in Middle and High Schools. A Report to Carnegie Corporation of New York. , 2007 .

[26]  C. Bazerman,et al.  Textual dynamics of the professions : historical and contemporary studies of writing in professional communities , 1991 .

[27]  M. Halliday Language as social semiotic: The social interpretation of language and meaning , 1976 .

[28]  C. Bazerman Speech Acts, Genres, and Activity Systems: How Texts Organize Activity and People , 2003 .

[29]  Chi-Fen Emily Chen,et al.  Beyond the Design of Automated Writing Evaluation: Pedagogical Practices and Perceived Learning Effectiveness in EFL Writing Classes. , 2008 .

[30]  Richard H. Haswell,et al.  Machine Scoring of Student Essays , 2006 .

[31]  M. Bakhtin,et al.  Speech genres and other late essays , 1986 .

[32]  B. Cope,et al.  The Powers of Literacy: A Genre Approach to Teaching Writing , 1993 .

[33]  Etienne Wenger,et al.  Situated Learning: Legitimate Peripheral Participation , 1991 .

[34]  J. Gee Semiotic Social Spaces and Affinity Spaces From The Age of Mythology to Today's Schools , 2005 .

[35]  Jill Burstein,et al.  Automated Essay Scoring : A Cross-disciplinary Perspective , 2003 .

[36]  Etienne Wenger,et al.  Communities of Practice: Learning, Meaning, and Identity , 1998 .

[37]  S. Heath Ways with Words: Language, Life and Work in Communities and Classrooms , 1983 .

[38]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[39]  Dc Washington National Governors Association and Chief Council of State School Officers. , 2010 .

[40]  David Finkelstein Handbook of Research on Writing , 2007 .

[41]  Semire Dikli,et al.  An Overview of Automated Scoring of Essays. , 2006 .

[42]  G. Hull,et al.  Locating the Semiotic Power of Multimodality , 2005 .

[43]  T. Landauer Automatic Essay Assessment , 2003 .

[44]  Patrick F. Reidy An Introduction to Latent Semantic Analysis , 2009 .

[45]  D. Barton,et al.  Beyond Communities of Practice: Language , 2005 .

[46]  Charles Bazerman,et al.  Handbook of research on writing : history, society, school, individual, text , 2007 .

[47]  Lev Vygotsky Mind in society , 1978 .