Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment

The quality of students’ writing skills continues to concern educators. Because writing is essential to success in both college and career, poor writing can have lifelong consequences. Writing is now primarily done digitally, but students receive limited explicit instruction in digital writing. This lack of instruction means that students fail to take advantage of the affordances of digital tools. The writing process is shaped by the tools used, which makes digital writing, to an extent, different from writing with pen and paper. To better understand students’ digital writing skills, we take advantage of the information provided by computer-based assessments—keyboard and mouse activity data. We examine the relationship between students’ use of the keyboard and mouse during the assessment and students’ writing achievement. Our data comes from the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth-grade students, we found that the number of keypresses had a distinct and direct effect on writing achievement scores, controlling for word count. We also identified several different patterns of keyboard and mouse activity on the computer-based NAEP assessment.

[1]  Russell G. Almond,et al.  A PRELIMINARY ANALYSIS OF KEYSTROKE LOG DATA FROM A TIMED WRITING TASK , 2012 .

[2]  Michael Russell,et al.  Does it Matter with What I Write? Comparing Performance on Paper, Computer and Portable Writing Devices , 2002 .

[3]  Joshua Wilson,et al.  Examining the Validity of Single-Occasion, Single-Genre, Holistically Scored Writing Assessments , 2012 .

[4]  Stephen Graham,et al.  Word processing programs and weaker writers/readers: a meta-analysis of research findings , 2012 .

[5]  Jeanne Wanzek,et al.  Writing evaluation: rater and task effects on the reliability of writing scores for children in Grades 3 and 4 , 2017, Reading and writing.

[6]  Kristen Purcell and Alan Heaps and Judy Buchanan and Linda Friedrich How Teachers Are Using Technology at Home and in Their Classrooms , 2013 .

[7]  Dorothy M. Chun,et al.  Technology in Language Use, Language Teaching, and Language Learning. , 2016 .

[8]  Randy Elliot Bennett,et al.  Online Assessment in Mathematics and Writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. , 2005 .

[9]  Kristen Purcell,et al.  The impact of digital tools on student writing and how writing is taught in schools , 2013 .

[10]  Jamal Abedi,et al.  The effects of prior computer use on computer-based writing: The 2011 NAEP writing assessment , 2016, Comput. Educ..

[11]  Amy Rathbun,et al.  Student Access to Digital Learning Resources outside of the Classroom. NCES 2017-098. , 2018 .

[12]  Jessica Singer Early,et al.  The Lifespan Development of Writing , 2018, Writing & Pedagogy.

[13]  S. Graham,et al.  A meta-analysis of writing instruction for adolescent students. , 2007 .

[14]  J. Wertsch Voices of the Mind: A Sociocultural Approach to Mediated Action , 1992 .

[15]  Nancy L. Allen,et al.  The NAEP 1998 Technical Report. , 2001 .

[16]  Colette Daiute,et al.  Physical and Cognitive Factors in Revising: Insights from Studies with Computers. , 1986 .

[17]  J. Wertsch From Social Interaction to Higher Psychological Processes. A Clarification and Application of Vygotsky’s Theory , 1979 .

[18]  Mark Warschauer,et al.  Laptops and Literacy: Learning in the Wireless Classroom , 2006 .

[19]  Lars R. Bergman,et al.  Studying Individual Development in an Interindividual Context: A Person-Oriented Approach , 2002 .

[20]  Stephen Graham,et al.  Explicitly Teaching Strategies, Skills, and Knowledge: Writing Instruction in Middle School Classrooms. , 2002 .

[21]  Andrew C. Porter,et al.  Common Core Standards , 2011 .

[22]  Mark Warschauer,et al.  Learning to compose digitally: the effect of prior computer use and keyboard activity on NAEP writing , 2019, Reading and Writing.

[23]  Randy Elliot Bennett,et al.  Does it Matter if I take My Writing Test on Computer? An Empirical Study of Mode Effects in NAEP , 2006 .

[24]  C. MacArthur Reflections on Research on Writing and Technology for Struggling Writers , 2009 .

[25]  Rebecca Zwick,et al.  Assessing the Dimensionality of NAEP Reading Data , 1987 .

[26]  Michael Russell,et al.  Testing Writing on Computers , 1997 .

[27]  D. Devoss,et al.  Because Digital Writing Matters: Improving Student Writing in Online and Multimedia Environments , 2010 .

[28]  Y. Engeström,et al.  Learning by expanding: An activity-theoretical approach to developmental research , 2014 .

[29]  Nancy L. Allen,et al.  Applying the Mantel-Haenszel Procedure to Complex Samples of Items , 1996 .

[30]  Edward Haertel,et al.  An Application of Latent Class Models to Assessment Data , 1984 .

[31]  Harold Wenglinsky Using Technology Wisely: The Keys to Success in Schools , 2005 .

[32]  J. Eccles,et al.  Expectancy-Value Theory of Achievement Motivation. , 2000, Contemporary educational psychology.

[33]  S. Engel Thought and Language , 1964 .

[34]  M. Halliday Towards a Language-Based Theory of Learning , 1993 .

[35]  Steve Graham,et al.  A Revised Writer(s)-Within-Community Model of Writing , 2018, Educational Psychologist.

[36]  Marilyn Cochran-Smith,et al.  Word Processing and Writing in Elementary Classrooms: A Critical Review of Related Literature , 1991 .

[37]  Lars R. Bergman,et al.  ROPstat: A General Statistical Package Useful for Conducting Person-Oriented Analyses , 2015 .

[38]  Gary A. Troia,et al.  Similarities and differences in constructs represented by U.S. States’ middle school writing tests and the 2007 National Assessment of Educational Progress writing assessment☆ , 2017 .

[39]  Jing Chen,et al.  Performance of Fourth-Grade Students in the 2012 NAEP Computer-Based Writing Pilot Assessment: Scores, Text Length, and Use of Editing Tools. Working Paper Series. NCES 2015-119. , 2015 .

[40]  Mary J. Schleppegrell,et al.  Understanding the Language Demands of Schooling: Nouns in Academic Registers , 2006 .

[41]  Mark Warschauer,et al.  Middle School Students’ Writing and Feedback in a Cloud-Based Classroom Environment , 2015, Technol. Knowl. Learn..

[42]  Charles A. MacArthur OVERCOMING BARRIERS TO WRITING: COMPUTER SUPPORT FOR BASIC WRITING SKILLS , 1999 .

[43]  Jacquelynne S. Eccles,et al.  Changes in academic motivation and self-perception during early adolescence. , 1990 .

[44]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[45]  A. Beaton,et al.  Chapter 1: Overview of the National Assessment of Educational Progress , 1992 .

[46]  Danielle S. McNamara,et al.  The Development of Writing Proficiency as a Function of Grade Level: A Linguistic Analysis , 2011 .

[47]  Eugene G. Johnson The Design of the National Assessment of Educational Progress , 1992 .

[48]  David D. Williams,et al.  Learning from Others: Service-Learning in Costa Rica and Indonesia. , 1997 .

[49]  Ruth Goldfine Making Word Processing More Effective in the Composition Classroom. , 2001 .

[50]  P. Deane WRITING ASSESSMENT AND COGNITION , 2011 .

[51]  Danielle S McNamara,et al.  Natural language processing in an intelligent writing strategy tutoring system , 2012, Behavior Research Methods.

[52]  Cynthia Greenleaf,et al.  Technological Indeterminacy , 1994 .

[53]  Gary L. Williamson,et al.  The Common Core State Standards’ Quantitative Text Complexity Trajectory , 2013 .

[54]  Qian Wang,et al.  The motivational landscape of early adolescence in the United States and China: a longitudinal investigation. , 2009, Child development.

[55]  D. McCutchen A capacity theory of writing: Working memory in composition , 1996 .

[56]  M. Bunge Mind and Society , 2010 .

[57]  Judith A. Langer,et al.  A Snapshot of Writing Instruction in Middle Schools and High Schools. , 2016 .

[58]  Richard J. Bernstein,et al.  The Varieties of Pluralism , 1985, American Journal of Education.

[59]  A. N. Leont’ev,et al.  Activity, consciousness, and personality , 1978 .