AcaWriter: A Learning Analytics Tool for Formative Feedback on Academic Writing

Written communication is an important skill across academia, the workplace, and civic participation. Effective writing incorporates instantiations of particular text structures - rhetorical moves - that communicate intent to the reader. These rhetorical moves are important across a range of academic styles of writing, including essays and research abstracts, as well as in forms of writing in which one reflects on learning gained through experience. However, learning how to effectively instantiate and use these rhetorical moves is a challenge. Moreover, educators often struggle to provide feedback supporting this learning, particularly at scale. Where effective support is provided, the techniques can be hard to share beyond single implementation sites. We address these challenges through the open-source AcaWriter tool, which provides feedback on rhetorical moves, with a design that allows feedback customization for specific contexts. We introduce three example implementations in which we have customized the tool and evaluated it with regard to user perceptions, and its impact on student writing. We discuss the tool's general theoretical background and provide a detailed technical account. We conclude with four recommendations that emphasize the potential of collaborative approaches in building, sharing and evaluating writing tools in research and practice.

[1]  E. Boyd,et al.  Reflective Learning , 1983 .

[2]  Bo Chang Reflection in Learning , 2019, Online Learning.

[3]  Jing Cai An exploratory study on an integrated genre-based approach for the instruction of academic lexical phrases , 2016 .

[4]  Yi Liu,et al.  Learning to write, reading to learn: genre, knowledge and pedagogy in the Sydney School , 2013 .

[5]  M. Ryan,et al.  Improving reflective writing in higher education: a social semiotic perspective , 2011 .

[6]  Abbie Brown,et al.  Design experiments: Theoretical and methodological challenges in creating complex interventions in c , 1992 .

[7]  Henk G. Schmidt,et al.  Writing to learn: can reflection journals be used to promote self-reflection and learning? , 2011 .

[8]  C. Lucas,et al.  Pharmacy Students’ Utilization of an Online Tool for Immediate Formative Feedback on Reflective Writing Tasks , 2019, American Journal of Pharmaceutical Education.

[9]  Simon Buckingham Shum,et al.  Reflective writing analytics for actionable feedback , 2017, LAK.

[10]  Laura K. Allen,et al.  Writing analytics literacy: bridging from research to practice , 2017, LAK.

[11]  Alison Lee,et al.  Bringing pedagogy to doctoral publishing , 2008 .

[12]  Maria Kuteeva,et al.  Graduate learners’ approaches to genre-analysis tasks: Variations across and within four disciplines , 2013 .

[13]  Claire Aitchison,et al.  ‘Tough love and tears’: learning doctoral writing in the sciences , 2012 .

[14]  M. Hough,et al.  Can a reflective rubric be applied consistently with raters globally? A study across three countries. , 2019, Currents in pharmacy teaching & learning.

[15]  Mary E. Ryan,et al.  Theorising a model for teaching and assessing reflective learning in higher education , 2013 .

[16]  Rod D. Roscoe,et al.  Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom , 2013 .

[17]  Elena Cotos,et al.  Automated writing evaluation for non-native speaker english academic writing: the case of iade and its formative feedback , 2010 .

[18]  Brian Paltridge,et al.  Thesis and Dissertation Writing in a Second Language , 2007, Journal of Second Language Writing.

[19]  Antonette Shibani,et al.  Turning the TAP on writing analytics , 2018 .

[20]  Mary Lea,et al.  Student writing in higher education: An academic literacies approach , 1998 .

[21]  Gregory F. Lorenz,et al.  A study of outcomes-oriented student reflection during internship: The integrated, coordinated, and reflection based model of learning and experiential education , 2010 .

[22]  Simon Buckingham Shum,et al.  Augmenting Formative Writing Assessment with Learning Analytics: A Design Abstraction Approach , 2018, ICLS.

[23]  Antje Proske,et al.  Digital support for academic writing: A review of technologies and pedagogies , 2019, Comput. Educ..

[24]  Ken Hyland,et al.  Genre pedagogy: Language, literacy and L2 writing instruction , 2007 .

[25]  E. Robinson,et al.  The Writing Experiences of Social Science Research Students. , 1992 .

[26]  Jill Burstein,et al.  Handbook of Automated Essay Evaluation Current Applications and New Directions , 2018 .

[27]  L. Gregg,et al.  Identifying the Organization of Writing Processes , 2016 .

[28]  N. Hatton,et al.  Reflection in teacher education: Towards definition and implementation , 1995 .

[29]  Alfred P. Rovai,et al.  The Internet and Higher Education , 2009 .

[30]  Peter Bruza,et al.  Towards the Discovery of Learner Metacognition From Reflective Writing , 2016, J. Learn. Anal..

[31]  Wayne Trotman Research Genres: Explorations and Applications , 2005 .

[32]  D. Boud Sustainable Assessment: Rethinking assessment for the learning society , 2000 .

[33]  Simon Buckingham Shum,et al.  Embracing imperfection in learning analytics , 2018, LAK.

[34]  Simon Knight,et al.  Design and implementation of a pedagogic intervention using writing analytics , 2017 .

[35]  Alma Fleet,et al.  Reflection: can we assess it? Should we assess it? , 1996 .

[36]  Simon Buckingham Shum,et al.  Contextualizable Learning Analytics Design: A Generic Model and Writing Analytics Evaluations , 2019, LAK.

[37]  Mary Ryan The Teachable Moment: The Washington Center Internship Program. , 1988 .

[38]  B. Cope,et al.  The Powers of Literacy: A Genre Approach to Teaching Writing , 1993 .

[39]  Laurence Anthony,et al.  Mover: a machine learning tool to assist in the reading and writing of technical papers , 2003 .

[40]  Shelley Burgin,et al.  Pedagogical Approaches that Facilitate Writing in Postgraduate Research Candidature in Science and Technology , 2011, Journal of University Teaching and Learning Practice.

[41]  Feng Wang,et al.  Design-based research and technology-enhanced learning environments , 2005 .

[42]  David R. Thomas,et al.  A General Inductive Approach for Analyzing Qualitative Evaluation Data , 2006 .

[43]  George Siemens,et al.  Penetrating the fog: analytics in learning and education , 2014 .

[44]  John M. Swales,et al.  Genre Analysis: English in Academic and Research Settings , 1993 .

[45]  M. Warschauer,et al.  Automated Writing Assessment in the Classroom , 2008 .

[46]  Stephanie Link,et al.  Understanding Graduate Writers’ Interaction with and Impact of the Research Writing Tutor during Revision , 2020, Journal of Writing Research.

[47]  Ulrike Goldschmidt,et al.  Academic Writing For Graduate Students , 2016 .

[48]  David F. Feldon,et al.  Faculty perceptions of common challenges encountered by novice doctoral writers , 2014 .

[49]  K. Eisenhardt Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[50]  Nancy Darcovich Literacy in the information age : final report of the International Adult Literacy Survey , 2000 .