Principled Assessment of Student Learning in High School Computer Science

As K-12 computer science (CS) initiatives scale throughout the U.S., educators face increasing pressure from their school systems to provide evidence about student learning on hard-to-measure CS outcomes. At the same time, researchers studying curriculum implementation and student learning want reliable measures of how students apply their CS knowledge. This paper describes a two-year validation study focused on end-of-unit and cumulative assessments for Exploring Computer Science, an introductory high school CS curriculum. To develop the assessments, we applied a principled methodology called Evidence-Centered Design (ECD) to (1) work with various stakeholders to identify the important computer science skills to measure, (2) map those skills to a model of evidence that can support inferences about those skills, and (3) develop assessment tasks that elicit that evidence. Using ECD, we created assessments that measure the practices of computational thinking, in contrast to assessments that only measure CS conceptual knowledge. We iteratively developed and piloted the assessments with 941 students over two years and collected three types of validity evidence based on contemporary psychometric standards: test content, internal structure, and student response processes. Results show that reliability was moderate to high for each of the unit assessments; the assessment tasks within each assessment are well aligned with each other and with the targeted learning goals; and average scores were in the 60 to 70 percent range. These results indicate that the assessments validly measure students' computational thinking practices covered in the introductory CS curriculum. We discuss the broader issues we faced of balancing the need to use the assessment results for evaluation and research, and demands from teachers for use in the classroom.