Design and First Results of a Psychometric Test for Measuring Basic Programming Abilities

We present the design of a test for measuring students' abilities concerning the application of control structures. Validated test instruments are a valuable tool for the evaluation of teaching both in a research setting as well as in a classroom setting. Our test is based on item-response-theory, in particular the Rasch model, and comprises a set of items all following the same format and using a simple, artificial programming language. We field-tested and modified the instrument in four iterations using only small samples and special statistical methods instead of the large samples usually required for IRT models. After the fourth iteration, the test has now reached a usable state. Based on the results, we were able to identify two misconceptions that are occurring very frequently in our test population - students of grade 7 to 10 in secondary schools.

[1]  Mark Guzdial,et al.  Assessing fundamental introductory computing concept knowledge in a language independent manner , 2010 .

[2]  Tim Futing Liao,et al.  Analysis of Multivariate Social Science Data , 2010 .

[3]  Almut E. Thomas,et al.  Skalen zur motivationalen Regulation beim Lernen von Schülerinnen und Schülern , 2011 .

[4]  G. H. Fischer,et al.  The linear logistic test model as an instrument in educational research , 1973 .

[5]  Ivo Ponocny,et al.  Nonparametric goodness-of-fit tests for the rasch model , 2001 .

[6]  Andreas Michael Mühling,et al.  Investigating Knowledge Structures in Computer Science Education , 2014 .

[7]  Leigh Ann Sudol-DeLyser,et al.  Analyzing test items: using item response theory to validate assessments , 2010, SIGCSE.

[8]  Cornelia M. Yoder,et al.  Nassi-Shneiderman charts an alternative to flowcharts for design , 1978 .

[9]  Ursula Fuller,et al.  Developing a computer science-specific learning taxonomy , 2007, ITiCSE-WGR '07.

[10]  Titus Winters,et al.  What do students know?: an outcomes-based assessment system , 2005, ICER '05.

[11]  Reinhold Hatzinger,et al.  Nonparametric tests for the Rasch model : explanation , development , and application of quasi-exact tests for small samples , 2013 .

[12]  Linda McIver,et al.  Research Agenda for Computer Science Education , 2001, PPIG.

[13]  Mark Guzdial,et al.  The FCS1: a language independent assessment of CS1 knowledge , 2011, SIGCSE.

[14]  Jens Bennedsen,et al.  A Competence Model for Object-Interaction in Introductory Programming , 2006, PPIG.

[15]  Louis V. DiBello,et al.  Guest Editors' Introduction and Overview: IRT‐Based Cognitive Diagnostic Models and Related Methods , 2007 .

[16]  Lauri Malmi,et al.  Characterizing research in computing education: a preliminary analysis of the literature , 2010, ICER '10.

[17]  Sally Fincher,et al.  Computer Science Education Research , 2004 .

[18]  Peter Hubwieser,et al.  Scratch vs. Karel: impact on learning outcomes and motivation , 2014, WiPSCE.

[19]  A. Wald.  Tests of statistical hypotheses concerning several parameters when the number of observations is large , 1943 .

[20]  Johannes Magenheim,et al.  Competence model for informatics modelling and system comprehension , 2013, 2013 IEEE Global Engineering Education Conference (EDUCON).

[21]  Cees A. W. Glas,et al.  Testing the Rasch Model , 1995 .

[22]  Richard E. Pattis,et al.  Karel the Robot: A Gentle Introduction to the Art of Programming , 1994 .

[23]  Valentina Dagiene,et al.  Bebras International Contest on Informatics and Computer Literacy: Criteria for Good Tasks , 2008, ISSEP.

[24]  Amruth N. Kumar,et al.  A study of the influence of code-tracing problems on code-writing skills , 2013, ITiCSE '13.

[25]  Kristy Elizabeth Boyer,et al.  A Practical Guide to Developing and Validating Computer Science Knowledge Assessments with Application to Middle School , 2015, SIGCSE.

[26]  Titus Winters,et al.  Closing the loop on test creation: a question assessment mechanism for instructors , 2006, SIGCSE '06.

[27]  E. B. Andersen,et al.  A goodness of fit test for the rasch model , 1973 .

[28]  Peter Hubwieser,et al.  Dimensions of Programming Knowledge , 2015, ISSEP.

[29]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.