Performance of Fourth-Grade Students in the 2012 NAEP Computer-Based Writing Pilot Assessment: Scores, Text Length, and Use of Editing Tools. Working Paper Series. NCES 2015-119.

This study examined whether or not fourth-graders could fully demonstrate their writing skills on the computer and factors associated with their performance on the National Assessment of Educational Progress (NAEP) computer-based writing assessment. The results suggest that highperforming fourth-graders (those who scored in the upper 20 percent in the computerand paperbased writing assessments on the NAEP) write better on the computer than on paper and can fully demonstrate their writing proficiency in a computer-based assessment. There are, however, indications that low-performing fourth-graders (i.e., those in the bottom 20 percent) and middle-performing fourth-graders (representing the middle 60 percent) may have performed less well on the computer than on paper. This suggests a potential differential effect of writing on the computer on writing performance that might have contributed to the widened achievement gap between highand non-high-performing (i.e., lowand middle-performing) students in the 2012 computer-based pilot assessment, compared to the 2010 paper-based pilot assessment. Unequal prior exposure to writing on the computer and preference for mode of writing (paper vs. computer) appear to be associated with this widening of the achievement gap. For example, about 94 percent of high-performing students in 2012 reported having access to the Internet at home, compared to about half (52 percent) of low-performing students. The score difference between those with access to the Internet at home and those without was substantive, with an effect size of 0.87. The statistics presented in this report are estimates based on a NAEP pilot assessment conducted with a nationally representative sample of 10,400 grade 4 students. As a pilot assessment, the sample was not adjusted to account for nonresponse bias. As a result, the sample may not completely represent the grade 4 population. Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment xxx This page intentionally left blank Introduction Chapter 1. 1. Background In 2012, the National Center for Education Statistics (NCES) administered the National Assessment of Educational Progress (NAEP) computer-based pilot writing assessment at grade 4. The key question that the pilot assessment was designed to answer was whether fourth-graders, especially low-performing students, could fully demonstrate their writing ability on a computer. Since this was a pilot assessment, no scale scores based on the item response theory (IRT) were produced. Instead, writing performance was reported using a 6-point scoring scale, where 1 represents the low end of performance and 6 represents the highest performance. Each sampled student was given two 30-minute tasks to complete and received a score that was computed as an average of their scores on the two tasks. The overall average score was about 3 points. Students used common word processing tools to compose and edit their responses to the writing assessment tasks. This NCES report provides the results of analyses that describe the length of their responses to these tasks as well as the frequency with which they used the editing tools. The sampled students were divided into three performance levels as follows: • Low: This level corresponds to those in the bottom 20 percent. The average score for students at this level is 1.5, which, according to the scoring rubric description, indicates writing skills in a low or marginal state. • Middle: This level corresponds to those in the middle 60 percent. The average score for students at this level is 3.0, which indicates writing skills in a developing state. • High: This level corresponds to those in the top 20 percent. The average score for students at this level is 4.8, which indicates close to competent writing skills. 2. Conceptual framework: overview Before conducting the analyses for this study, it was first necessary to develop a conceptual framework of the relationships that exist between fourth-graders’ writing performance, their keyboarding skills and use of editing tools (referred to here as facilitative skills), and relevant contextual factors. The diagram in figure 1 displays writing performance as a dependent variable, with three associated factors—text length, preference for mode of writing (paper vs. computer), and the use of editing tools. Prior exposure to writing on the computer is assumed to be positively correlated with all three factors. In this diagram, the direction of the arrows indicates how this study conceptualizes the relationship of writing performance with various relevant factors that act as predictors, but not as causal factors. Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment 1 Chapter