Design and validation of learning trajectory-based assessments for computational thinking in upper elementary grades

ABSTRACT Background & Context: We describe the rationale, design, and initial validation of computational thinking (CT) assessments to pair with curricular lessons that integrate fractions and CT. Objective: We used cognitive models of CT (learning trajectories; LTs) to design assessments and obtained evidence to support a validity argument Method: We used the LTs and evidence-centered design to develop assessments that 144 Grade 3 and Grade 4 elementary students completed following the integrated instruction. We analyzed data using multiple psychometric approaches. Findings: The design approach and data analysis suggest that the assessments are well-suited to evaluate students’ CT knowledge, skills and abilities across multiple LTs. Implications: We show how to use LTs to design assessments that can yield valid inferences about students’ CT competencies; these methods can be adopted and extended by others to create additional assessments that can advance computer science education.

[1]  Jere Confrey,et al.  Learning over Time: Learning Trajectories in Mathematics Education. , 2014 .

[2]  Aman Yadav,et al.  Computational thinking in compulsory education: Towards an agenda for research and practice , 2015, Education and Information Technologies.

[3]  M. Kane.  The Argument-Based Approach to Validation , 1990 .

[4]  Michael S. Horn,et al.  Defining Computational Thinking for Mathematics and Science Classrooms , 2016 .

[5]  R. Shavelson,et al.  On the evaluation of systemic science education reform: Searching for instructional sensitivity , 2002 .

[6]  Michael Gr. Voskoglou,et al.  Problem Solving and Computational Thinking in a Learning Environment , 2012, ArXiv.

[7]  R. Glaser,et al.  Knowing What Students Know: The Science and Design of Educational Assessment , 2001 .

[8]  Brian D. Gane,et al.  Measuring What Matters: Using Technology to Assess Multidimensional Learning. , 2018 .

[9]  Diana Franklin,et al.  Action Fractions: The Design and Pilot of an Integrated Math+CS Elementary Curriculum Based on Learning Trajectories , 2021, SIGCSE.

[10]  James W. Pellegrino,et al.  Cambridge Handbook of Engineering Education Research: The Science and Design of Assessment in Engineering Education , 2014 .

[11]  Barbara M. Olds,et al.  Rigorous Methodology for Concept Inventory Development: Using the 'Assessment Triangle' to Develop and Test the Thermal and Transport Science Concept Inventory (TTCI)* , 2011 .

[12]  Chris Stephenson,et al.  Bringing computational thinking to K-12: what is Involved and what is the role of the computer science education community? , 2011, INROADS.

[13]  Educational Evaluation.  Standards for Educational and Psychological Testing , 1999 .

[14]  Eric Rosenbaum,et al.  Scratch: programming for all , 2009, Commun. ACM.

[15]  Gregorio Robles,et al.  Dr. Scratch: Automatic Analysis of Scratch Projects to Assess and Foster Computational Thinking , 2015 .

[16]  John Maloney,et al.  The Scratch Programming Language and Environment , 2010, TOCE.

[17]  Maya Israel,et al.  Supporting all learners in school-wide computational thinking: A cross-case qualitative analysis , 2015, Comput. Educ..

[18]  V. Shute,et al.  Demystifying computational thinking , 2017 .

[19]  W. R. Dharmathilaka.  වර්ණ සන්නිවේදනය හා සංඥාර්ථ : ශ්රී ලාංකේය සංස්කෘතික වර්ණ සන්නිවේදනයේ පරිණාමය පිළිබඳ අධ්යයනයක් (Unpublished doctoral dissertation) , 2016 .

[20]  P. Mair,et al.  Extended Rasch Modeling: The eRm Package for the Application of IRT Models in R , 2007 .

[21]  A. Viera,et al.  Understanding interobserver agreement: the kappa statistic. , 2005, Family medicine.

[22]  Aman Yadav,et al.  Computational Thinking for All: Pedagogical Approaches to Embedding 21st Century Problem Solving in K-12 Classrooms , 2016 .

[23]  Arnold Zellner,et al.  Introduction to Measurement with Theory , 2009 .

[24]  Yue Yin,et al.  Assessing computational thinking: A systematic review of empirical studies , 2020, Comput. Educ..

[25]  Susan R. Goldman,et al.  A Framework for Conceptualizing and Evaluating the Validity of Instructionally Relevant Assessments , 2016 .

[26]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[27]  Marcos Romn-Gonzlez,et al.  Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test , 2017, Comput. Hum. Behav..

[28]  Jalal Nouri,et al.  A systematic review of learning computational thinking through Scratch in K-9 , 2019, Comput. Educ..

[29]  A. Baroody,et al.  Efficacy of a learning trajectory approach compared to a teach-to-target approach for addition and subtraction , 2020 .

[30]  Robert J. Mislevy,et al.  Implications of Evidence‐Centered Design for Educational Testing , 2007 .

[31]  Marie Bienkowski,et al.  Leveraging Evidence-Centered Design to Develop Assessments of Computational Thinking Practices , 2019, International Journal of Testing.

[32]  John Harrison,et al.  Computational Thinking: A Digital Age Skill for Everyone. , 2011 .

[33]  Beth Simon,et al.  Evaluating a new exam question: Parsons problems , 2008, ICER '08.

[34]  Linda M. Seiter,et al.  Modeling the learning progressions of computational thinking of primary grade students , 2013, ICER.

[35]  Marina Umaschi Bers,et al.  Computational thinking and tinkering: Exploration of an early childhood robotics curriculum , 2014, Comput. Educ..

[36]  Brian D. Gane,et al.  Understanding Students' Computational Thinking through Cognitive Interviews: A Learning Trajectory-based Analysis , 2020, SIGCSE.

[37]  S. Selcen Guzey,et al.  A Study of STEM Assessments in Engineering, Science, and Mathematics for Elementary and Middle School Students , 2015 .

[38]  Kerrie A. Douglas,et al.  Validity: Meaning and Relevancy in Assessment for Engineering Education Research , 2015 .

[39]  Gregorio Robles,et al.  Combining Assessment Tools for a Comprehensive Evaluation of Computational Thinking Interventions , 2019, Computational Thinking Education.

[40]  Thomas W. Price,et al.  Comparing Textual and Block Interfaces in a Novice Programming Environment , 2015, ICER.

[41]  June Ahn,et al.  Equity in the Who, How and What of Computer Science Education: K12 School District Conceptualizations of Equity in ‘CS for All’ Initiatives , 2019, 2019 Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT).

[42]  Fred Martin,et al.  Integrating computational thinking across the K--8 curriculum , 2014, Inroads.

[43]  Ty W. Boyer,et al.  Child proportional scaling: is 1/3=2/6=3/9=4/12? , 2012, Journal of experimental child psychology.

[44]  Lyn D. English,et al.  Advancing Elementary and Middle School STEM Education , 2017 .

[45]  Descriptors Educational,et al.  of Educational Measurement , 1988 .

[46]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[47]  F. Baker.  The basics of item response theory , 1985 .

[48]  Diana Franklin,et al.  Decomposition: A K-8 Computational Thinking Learning Trajectory , 2018, ICER.

[49]  MartinFred,et al.  Integrating computational thinking across the K--8 curriculum , 2014 .

[50]  Brian D. Gane,et al.  Contemporary Methods of Assessing Integrated STEM Competencies , 2020, Handbook of Research on STEM Education.

[51]  Diana Franklin,et al.  A K-8 Debugging Learning Trajectory Derived from Research Literature , 2019, SIGCSE.

[52]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[53]  A Learning Trajectory for Variables Based in Computational Thinking Literature: Using Levels of Thinking to Develop Instruction , 2020, Computer Science Education.

[54]  Diana Franklin,et al.  K-8 Learning Trajectories Derived from Research Literature: Sequence, Repetition, Conditionals , 2017, ICER.

[55]  Jeannette M. Wing.  An introduction to computer science for non-majors using principles of computation , 2007, SIGCSE.

[56]  Pep Simo,et al.  Low-Cost educational videos' for engineering students: a new concept based on video streaming and Youtube channels , 2011 .

[57]  Barbara Ericson,et al.  Solving parsons problems versus fixing and writing code , 2017, Koli Calling.