A Practical Guide to Developing and Validating Computer Science Knowledge Assessments with Application to Middle School

Knowledge assessment instruments, or tests, are commonly created by faculty in classroom settings to measure student knowledge and skill. Another crucial role for assessment instruments is in gauging student learning in response to a computer science education research project, or intervention. In an increasingly interdisciplinary landscape, it is crucial to validate knowledge assessment instruments, yet developing and validating these tests for computer science poses substantial challenges. This paper presents a seven-step approach to designing, iteratively refining, and validating knowledge assessment instruments designed not to assign grades but to measure the efficacy or promise of novel interventions. We also detail how this seven-step process is being instantiated within a three-year project to implement a game-based learning environment for middle school computer science. This paper serves as a practical guide for adapting widely accepted psychometric practices to the development and validation of computer science knowledge assessments to support research.

[1]  Timothy R. Hinkin.  A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires , 1998 .

[2]  Kristy Elizabeth Boyer,et al.  CS principles goes to middle school: learning how to teach "Big Data" , 2014, SIGCSE.

[3]  Athanasios V. Vasilakos,et al.  Big data: From beginning to future , 2016, Int. J. Inf. Manag..

[4]  Timothy Shanahan,et al.  Common Core State Standards: A New Role for Writing. , 2015 .

[5]  Diana Franklin,et al.  Assessment of computer science learning in a scratch-based outreach program , 2013, SIGCSE '13.

[6]  Jill Denner,et al.  The fairy performance assessment: measuring computational thinking in middle school , 2012, SIGCSE '12.

[7]  De Ayala,et al.  The Theory and Practice of Item Response Theory , 2008 .

[8]  Daryl J. D'Souza,et al.  Instructor perspectives of multiple-choice questions in summative assessment for novice programmers , 2010, Comput. Sci. Educ..

[9]  Mark Guzdial,et al.  The FCS1: a language independent assessment of CS1 knowledge , 2011, SIGCSE.

[10]  Jeffrey D. Karpicke,et al.  The Power of Testing Memory Basic Research and Implications for Educational Practice , 2006 .

[11]  Leigh Ann Sudol-DeLyser,et al.  Analyzing test items: using item response theory to validate assessments , 2010, SIGCSE.

[12]  Gerald V. Barrett,et al.  Validity of Personnel Decisions: A Conceptual Analysis of the Inferential and Evidential Bases , 1989 .

[13]  Eli P. Cox,et al.  The optimal number of response alternatives for a scale: A review. , 1980 .

[14]  Mark Guzdial,et al.  Developing a validated assessment of fundamental CS1 concepts , 2010, SIGCSE.

[15]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[16]  David Broman,et al.  An assessment model for large project courses , 2014, SIGCSE '14.