New Horizons in the Assessment of Computer Science at School and Beyond: Leveraging on the ViVA Platform

A revolution is taking place across Europe and worldwide in how we teach our children about computing, in primary and secondary school. Out goes ICT and how to use Microsoft Office; in comes coding and computer science. Assessment has a crucial role to play in this revolution. If teachers use low-quality assessment instruments we will end-up teaching the wrong subject; and viceversa. This paper reviews the state of the field, and makes concrete, achievable proposals for developing shared, high quality assessments for computer science. Central to this proposal is the collaborative platform VIVA (the Vilnius collaboratively coded and Validated computer science questions/tasks for Assess- ment). Two requirements are key to VIVA: 1) support for multiple competency frameworks, so that the contributors can meta-tag resources with respect to the framework they are most familiar with; and 2) support for crowdsourcing the validation of each question/task and its mapping to competencies. The use of a taxonomy of questions/tasks type that has been mapped to computational thinking concepts and to a competency framework is proposed. Some seed questions are already available in the online platform prototype, and various supporters have granted permission to use large questions banks. The design requirements of a full implementation of the VIVA platform for a modern and effective approach to assessment including support for digital badges, are outlined; and some preliminary results from a survey administered to the initial contributors to VIVA are presented.

[1]  Henry Ernest Dudeney,et al.  The Canterbury Puzzles , 1907 .

[2]  Katerina Ananiadou,et al.  21st Century Skills and Competences for New Millennium Learners in OECD Countries , 2009 .

[3]  David Wolber,et al.  Teaching the CS principles curriculum with App Inventor (abstract only) , 2013, SIGCSE '13.

[4]  Michail N. Giannakos,et al.  Computing education in K-12 schools: A review of the literature , 2015, 2015 IEEE Global Engineering Education Conference (EDUCON).

[5]  Sridhar Iyer,et al.  Question-Posing strategies used by students for exploring Data Structures , 2015, ITiCSE.

[6]  Eric Rosenbaum,et al.  Scratch: programming for all , 2009, Commun. ACM.

[7]  Ari Korhonen,et al.  How (not) to introduce badges to online exercises , 2014, SIGCSE.

[8]  Simone Palazzo,et al.  An innovative web-based collaborative platform for video annotation , 2014, Multimedia Tools and Applications.

[9]  Tony Clear,et al.  An introduction to program comprehension for computer science educators , 2010, ITiCSE-WGR '10.

[10]  Peter Hubwieser,et al.  How teachers in different educational systems value central concepts of computer science , 2012, WiPSCE.

[11]  Peter J. Denning,et al.  The science in computer science , 2013, CACM.

[12]  Mordechai Ben-Ari,et al.  Learning computer science concepts with scratch , 2010, ICER '10.

[13]  David C. Webb,et al.  Scalable Game Design , 2015, ACM Trans. Comput. Educ..

[14]  Rebecca Ferguson,et al.  Open University: Innovating Pedagogy 2013 , 2014 .

[15]  Daniel Zingaro,et al.  Reviewing CS1 exam question content , 2011, SIGCSE '11.

[16]  Christina Dörge,et al.  Competencies and Skills: Filling Old Skins with New Wine , 2010, Key Competencies in the Knowledge Society.

[17]  Leigh Ann Sudol-DeLyser,et al.  Code comprehension problems as learning events , 2012, ITiCSE '12.

[18]  George E. DeBoer,et al.  Assessment Linked to Science Learning Goals: Probing Student Thinking Through Assessment , 2008 .

[19]  Joyce Malyn-Smith,et al.  Computational thinking for youth in practice , 2011, INROADS.

[20]  Carl E. Wieman,et al.  Editorial for special issue on concept inventories in computing , 2014, Comput. Sci. Educ..

[21]  Shuchi Grover,et al.  Building a virtual community of practice for K-12 CS teachers , 2014, CACM.

[22]  Petri Ihantola,et al.  Review of recent systems for automatic assessment of programming assignments , 2010, Koli Calling.

[23]  J. Delors.  Learning: The Treasure within. Report to UNESCO of the International Commission on Education for the twenty-first-Century, Paris UNESCO 1996 , 1996 .

[24]  Mark Guzdial,et al.  Setting the stage for computing curricula 2013: computer science -- report from the ACM/IEEE-CS joint task force , 2011, SIGCSE.

[25]  David Weintrop,et al.  Robobuilder: a computational thinking game (abstract only) , 2013, SIGCSE '13.

[26]  Helen H. Hu,et al.  Using POGIL to help students learn to program , 2013, TOCE.

[27]  Shun-Shii Lin,et al.  Optimal algorithms for constructing knight's tours on arbitrary n?m chessboards , 2005, Discret. Appl. Math..

[28]  Riina Vuorikari,et al.  European Schoolnet: enabling school networking , 2009 .

[29]  Craig S. Miller,et al.  Introduction to the Special Issue on Web Development , 2015, ACM Trans. Comput. Educ..

[30]  Katie Davis,et al.  Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators , 2015, Comput. Educ..

[31]  R. Hake.  Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses , 1998 .

[32]  Roy D. Pea,et al.  Assessing computational learning in K-12 , 2014, ITiCSE '14.

[33]  Hadi Partovi,et al.  A comprehensive effort to expand access and diversity in computer science , 2015, Inroads.

[34]  Robb Lindgren,et al.  Using badges for shaping interactions in online learning environments , 2012, 2012 IEEE International Professional Communication Conference.

[35]  Cynthia Selby,et al.  Developing Computational Thinking in the Classroom: A Framework , 2014 .

[36]  Michael Goldweber,et al.  A framework for enhancing the social good in computing education: a values approach , 2012, ITiCSE-WGR '12.

[37]  Cindy C. Parman,et al.  To code, or not to code? , 2003, The Journal of oncology management : the official journal of the American College of Oncology Administrators.

[38]  Shitanshu Mishra.  Developing students' problem-posing skills , 2014, ICER '14.

[39]  Robert McCartney,et al.  Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer , 2010, SGCS.

[40]  Christina Dörge.  A methodological approach to key competencies in informatics , 2014, ITiCSE '14.

[41]  Nickolas J. G. Falkner,et al.  "Whither, badges?" or "wither, badges!": a metastudy of badges in computer science education to clarify effects, significance and influence , 2014, Koli Calling.

[42]  Daniela Giordano,et al.  Quizly: A live coding assessment platform for App Inventor , 2015, 2015 IEEE Blocks and Beyond Workshop (Blocks and Beyond).

[43]  Paul Ashwin,et al.  Teaching to the test , 2007 .

[44]  Sarah Hug,et al.  Cultivating a K12 computer science community: a case study , 2013, SIGCSE '13.

[45]  Stephen H. Edwards,et al.  The Canterbury QuestionBank: building a repository of multiple-choice CS1 and CS2 questions , 2013, ITiCSE -WGR '13.

[46]  Valentina Dagiene.  Sustaining Informatics Education by Contests , 2010, ISSEP.

[47]  Daniela Giordano,et al.  Teaching algorithms: Visual language vs flowchart vs textual language , 2015, 2015 IEEE Global Engineering Education Conference (EDUCON).

[48]  K. Allen,et al.  The Statistics Concept Inventory: Development and Analysis of a Cognitive Assessment Instrument in Statistics , 2006 .

[49]  Yigal Attali,et al.  Gamification in assessment: Do points affect test performance? , 2015, Comput. Educ..

[50]  Vladan Devedzic,et al.  Open Badges: Novel Means to Motivate, Scaffold and Recognize Learning , 2015, Technol. Knowl. Learn..

[51]  Ashraf Elnagar,et al.  An Intelligent Assessment Tool for Students’ Java Submissions in Introductory Programming Courses , 2012 .

[52]  Ursula Fuller,et al.  Perspectives on developing and assessing professional values in computing , 2010, SGCS.

[53]  Paul Denny,et al.  Generating Practice Questions as a Preparation Strategy for Introductory Programming Exams , 2015, SIGCSE.

[54]  David Wolber,et al.  Mobile computer science principles: a professional development sampler for teachers (abstract only) , 2014, SIGCSE '14.

[55]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[56]  Allison Elliott Tew,et al.  A fresh look at novice programmers' performance and their teachers' expectations , 2013, ITiCSE -WGR '13.

[57]  Simon,et al.  Academic integrity policies in a computing education context , 2012, ITiCSE-WGR '12.

[58]  James M. Boyle,et al.  A systematic literature review of empirical evidence on computer games and serious games , 2012, Comput. Educ..

[59]  Emily M. Geraghty Ward,et al.  Revisiting the Geoscience Concept Inventory: A call to the community , 2011 .

[60]  Scott Grissom.  Introduction to special issue on alternatives to lecture in the computer science classroom , 2013, TOCE.

[61]  John Hamer,et al.  The PeerWise system of student contributed assessment questions , 2008, ACE '08.

[62]  Jan Vahrenhold,et al.  Developing and validating test items for first-year computer science courses , 2014, Comput. Sci. Educ..

[63]  Tyler A. Kummer,et al.  Teaching to the Test…or Testing to Teach: Exams Requiring Higher Order Thinking Skills Encourage Greater Conceptual Understanding , 2014 .

[64]  D. Hestenes,et al.  Force concept inventory , 1992 .

[65]  Linda M. Seiter,et al.  Modeling the learning progressions of computational thinking of primary grade students , 2013, ICER.

[66]  J. Voogt,et al.  A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies , 2012 .

[67]  David Wolber,et al.  Events-first programming in APP inventor , 2014 .

[68]  Sue Bennett,et al.  Technology tools to support learning design: Implications derived from an investigation of university teachers' design practices , 2015, Comput. Educ..

[69]  Cynthia Selby,et al.  Evidence of assessing computational thinking , 2014 .

[70]  Kirk Allen,et al.  The statistics concept inventory: a pilot study , 2003, 33rd Annual Frontiers in Education, 2003. FIE 2003..

[71]  John D. Bransford,et al.  New Assessments and Environments for Knowledge Building , 2012 .

[72]  Briana B. Morrison,et al.  Khan academy gamifies computer science , 2014, SIGCSE.

[73]  Christian Spannagel,et al.  Empirical Foundation of Central Concepts for Computer Science Education , 2008, JERC.

[74]  Ari Korhonen,et al.  The Effect of Achievement Badges on Students' Behavior: An Empirical Study in a University-Level Computer Science Course , 2015, Int. J. Emerg. Technol. Learn..

[75]  Michael Goldweber,et al.  Non-programming resources for an introduction to CS: a collection of resources for the first courses in computer science , 2001, SGCS.

[76]  Alexander Repenning,et al.  Real Time Assessment of Computational Thinking , 2014, 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[77]  Sally Fincher,et al.  Identifying threshold concepts: from dead end to a new direction , 2013, ICER.

[78]  R. Beeby,et al.  Assessment Integration in Serious Games , 2014 .

[79]  Dieter Klaudt,et al.  Empirical Determination of Competence Areas to Computer Science Education , 2014 .

[80]  Marina Umaschi Bers,et al.  Bringing Grades K-5 to the Mainstream of Computer Science Education , 2015, SIGCSE.

[81]  David Broman,et al.  An assessment model for large project courses , 2014, SIGCSE '14.

[82]  Leen-Kiat Soh,et al.  Concept inventories in computer science for the topic discrete mathematics , 2006, ACM SIGCSE Bull..

[83]  Valentina Dagiene,et al.  Bebras International Contest on Informatics and Computer Literacy: Criteria for Good Tasks , 2008, ISSEP.

[84]  Robert McCartney,et al.  Editorial: Computing Education in (K-12) Schools from a Cross-National Perspective , 2014, TOCE.

[85]  Jennifer K Knight,et al.  The Genetics Concept Assessment: a new concept inventory for gauging student understanding of genetics. , 2008, CBE life sciences education.

[86]  Stephen H. Edwards,et al.  Developing a common format for sharing programming assignments , 2008, ACM SIGCSE Bull..

[87]  Daryl J. D'Souza,et al.  How difficult are exams?: a framework for assessing the complexity of introductory programming exams , 2013, ACE '13.

[88]  Kristy Elizabeth Boyer,et al.  A Practical Guide to Developing and Validating Computer Science Knowledge Assessments with Application to Middle School , 2015, SIGCSE.

[89]  Timo Poranen,et al.  Reasoning on Children's Cognitive Skills in an Informatics Contest: Findings and Discoveries from Finland, Lithuania, and Sweden , 2014, ISSEP.

[90]  Cynthia Taylor,et al.  Computer science concept inventories: past and future , 2014, Comput. Sci. Educ..

[91]  Liz Bacon,et al.  A Serious Game for Developing Computational Thinking and Learning Introductory Computer Programming , 2012 .

[92]  Vladan Devedžić,et al.  Developing Open Badges: a comprehensive approach , 2015 .

[93]  Chinma Uche,et al.  Analyzing Year One of a CS Principles PD Project , 2015, SIGCSE.

[94]  Roy D. Pea,et al.  Designing for deeper learning in a blended computer science course for middle school students , 2015, Comput. Sci. Educ..

[95]  Diana Franklin,et al.  Hairball: lint-inspired static analysis of scratch projects , 2013, SIGCSE '13.

[96]  Lasse HAKULINEN,et al.  Survey on Informatics Competitions: Developing Tasks , 2011 .

[97]  Alexander Repenning.  Agentsheets: a tool for building domain-oriented visual programming environments , 1993, CHI '93.

[98]  Rachel Cardell-Oliver,et al.  Motivating all our students? , 2011, ITiCSE-WGR.

[99]  Beth Simon,et al.  Peerwise: replication study of a student-collaborative self-testing web service in a u.s. setting , 2010, SIGCSE.

[100]  Daniela Giordano,et al.  Use of cutting edge educational tools for an initial programming course , 2014, 2014 IEEE Global Engineering Education Conference (EDUCON).

[101]  Pierluigi Crescenzi,et al.  Adapting moodle to better support CS education , 2010, ITiCSE-WGR '10.

[102]  Juho Hamari,et al.  Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification , 2014, 2014 47th Hawaii International Conference on System Sciences.

[103]  J. Herman,et al.  Defining Twenty-First Century Skills , 2012 .

[104]  Gregorio Robles,et al.  Automatic detection of bad programming habits in scratch: A preliminary study , 2014, 2014 IEEE Frontiers in Education Conference (FIE) Proceedings.

[105]  Dario Malchiodi,et al.  Informatics Education in Italian Secondary Schools , 2014, TOCE.

[106]  Marie Nordström,et al.  An evaluation of object oriented example programs in introductory programming textbooks , 2010, SGCS.

[107]  David J. McArthur,et al.  Metadata lessons from the iLumina digital library , 2005, CACM.

[108]  Amber Settle,et al.  Computational Thinking in K-9 Education , 2014, ITiCSE-WGR.

[109]  Michelle K. Smith,et al.  Why Peer Discussion Improves Student Performance on In-Class Concept Questions , 2009, Science.

[110]  Mark Guzdial,et al.  Developing a validated assessment of fundamental CS1 concepts , 2010, SIGCSE.

[111]  Susan M. Mitchell,et al.  Assessing the Value of Computer Science Course Material Repositories , 2006, 19th Conference on Software Engineering Education and Training Workshops (CSEETW'06).

[112]  W. Robinson,et al.  An Inventory for Alternate Conceptions among First-Semester General Chemistry Students , 2002 .

[113]  Orit Hazzan,et al.  Guide to Teaching Computer Science: An Activity-Based Approach , 2011 .

[114]  Gregorio Robles,et al.  The Europe Code Week (CodeEU) initiative shaping the skills of future engineers , 2015, 2015 IEEE Global Engineering Education Conference (EDUCON).

[115]  Jean Gordon,et al.  Key Competences in Europe: Opening Doors for Lifelong Learners Across the School Curriculum and Teacher Education , 2009 .

[116]  Daniel Zingaro,et al.  Importance of early performance in CS1: two conflicting assessment stories , 2014, SIGCSE '14.

[117]  Herman Lam,et al.  Setting the stage for CE2016: A revised body of knowledge , 2014, 2014 IEEE Frontiers in Education Conference (FIE) Proceedings.

[118]  Beth Simon,et al.  Evaluating a new exam question: Parsons problems , 2008, ICER '08.

[119]  Michail N. Giannakos,et al.  How to Implement Rigorous Computer Science Education in K-12 Schools? Some Answers and Many Questions , 2015, ACM Trans. Comput. Educ..

[120]  Richard C. H. Connor,et al.  Code or (not code): separating formal and natural language in CS education , 2014, WiPSCE.

[121]  Sébastien Combéfis,et al.  Organising National Olympiads in Informatics : a Review of Selection Processes , Trainings and Promotion Activities , 2015 .

[122]  B. Hufnagel,et al.  Development of the Astronomy Diagnostic Test , 2001 .

[123]  Diana Franklin,et al.  Assessment of computer science learning in a scratch-based outreach program , 2013, SIGCSE '13.

[124]  Cynthia Bailey Lee,et al.  Can peer instruction be effective in upper-division computer science courses? , 2013, TOCE.

[125]  Peter Wentworth,et al.  Computational thinking in educational activities: an evaluation of the educational game light-bot , 2013, ITiCSE '13.

[126]  Paul Denny.  Motivating online collaborative learning , 2010, ITiCSE '10.

[127]  Neil Brown,et al.  Repositories of teaching material and communities of use: nifty assignments and the greenroom , 2010, ICER '10.

[128]  Sally Fincher,et al.  Machines for Thinking , 2010, TOCE.

[129]  Roland Mittermeir,et al.  Computer science/informatics in secondary education , 2011, ITiCSE-WGR.

[130]  J. Carter,et al.  How shall we assess this? , 2003, ITiCSE-WGR '03.

[131]  Ernesto Cuadros-Vargas,et al.  ACM/IEEE-CS computer science curriculum 2013: reviewing the ironman report , 2013, SIGCSE '13.