On the Automatic Assessment of Computational Thinking Skills: A Comparison with Human Experts

Programming and computational thinking skills are promoted in schools worldwide. However, there is still a lack of tools that assist learners and educators in the assessment of these skills. We have implemented an assessment tool, called Dr. Scratch, that analyzes Scratch projects with the aim to assess the level of development of several aspects of computational thinking. One of the issues to address in order to show its validity is to compare the (automatic) evaluations provided by the tool with the (manual) evaluations by (human) experts. In this paper we compare the assessments provided by Dr. Scratch with over 450 evaluations of Scratch projects given by 16 experts in computer science education. Our results show strong correlations between automatic and manual evaluations. As there is an ample debate among educators on the use of this type of tools, we discuss the implications and limitations, and provide recommendations for further research.

[1]  Felienne Hermans,et al.  Do code smells hamper novice programming? A controlled experiment on Scratch programs , 2016, 2016 IEEE 24th International Conference on Program Comprehension (ICPC).

[2]  Chris Stephenson,et al.  Bringing computational thinking to K-12: what is Involved and what is the role of the computer science education community? , 2011, INROADS.

[3]  Gregorio Robles,et al.  Automatic detection of bad programming habits in scratch: A preliminary study , 2014, 2014 IEEE Frontiers in Education Conference (FIE) Proceedings.

[4]  Seymour Papert,et al.  Mindstorms: Children, Computers, and Powerful Ideas , 1981 .

[5]  Murray Hill,et al.  Lint, a C Program Checker , 1978 .

[6]  Linda M. Seiter,et al.  Modeling the learning progressions of computational thinking of primary grade students , 2013, ICER.

[7]  Leon J. Osterweil,et al.  Omega -- A Data Flow Analysis Tool for the C Programming Language ; CU-CS-217-82 , 1982 .

[8]  Ricardo Olmos,et al.  Transforming LSA space dimensions into a rubric for an automatic assessment and feedback system , 2016, Inf. Process. Manag..

[9]  Gregorio Robles,et al.  Dr. Scratch: Automatic Analysis of Scratch Projects to Assess and Foster Computational Thinking , 2015 .

[10]  Gregorio Robles,et al.  Comparing computational thinking development assessment scores with software complexity metrics , 2016, 2016 IEEE Global Engineering Education Conference (EDUCON).

[11]  Jeannette M. Wing.  An introduction to computer science for non-majors using principles of computation , 2007, SIGCSE.

[12]  Diana Franklin,et al.  Hairball: lint-inspired static analysis of scratch projects , 2013, SIGCSE '13.

[13]  Thomas Connolly,et al.  Evaluation of Computer Games Developed by Primary School Children to Gauge Understanding of Programming Concepts , 2012 .

[14]  Felienne Hermans,et al.  How Kids Code and How We Know: An Exploratory Study on the Scratch Repository , 2016, ICER.

[15]  Matt MacLaurin,et al.  Kodu: end-user programming and design for games , 2009, FDG.

[16]  Diana Franklin,et al.  Assessment of computer science learning in a scratch-based outreach program , 2013, SIGCSE '13.

[17]  Kevin D. Carlson,et al.  Understanding the Impact of Convergent Validity on Research Results , 2012 .

[18]  Leon J. Osterweil,et al.  Omega—A Data Flow Analysis Tool for the C Programming Language , 1985, IEEE Transactions on Software Engineering.

[19]  Eric Rosenbaum,et al.  Scratch: programming for all , 2009, Commun. ACM.

[20]  Randy Pausch,et al.  Alice: a 3-D tool for introductory programming concepts , 2000 .

[21]  Amy K. Hoover,et al.  Assessing Computational Thinking in Students' Game Designs , 2016, CHI PLAY.

[22]  Alexander Repenning,et al.  Towards the Automatic Recognition of Computational Thinking for Adaptive Visual Language Learning , 2010, 2010 IEEE Symposium on Visual Languages and Human-Centric Computing.