Aggregate Compilation Behavior: Findings and Implications from 27,698 Users

The error quotient (EQ) was first reported in 2006 as a behavioral measure of novice programmers. The EQ scores how well students deal with correcting syntax errors (or not) in their programs. The original studies were carried out on data collected using BlueJ, a pedagogic Java programming environment; today, newly installed instances of BlueJ capture data similar to these early studies automatically, meaning data regarding nearly 2 million programmers is captured every year by the Blackbox project. In this paper, we apply Jadud's original error quotient algorithm to this new, massive data set, and discuss our results and analysis in light of related work. Further, we consider the implications of our findings for researchers and educators in applying the EQ to 27,698 users in 10 different countries during the fall term of 2013.

[1]  Ryan Shaun Joazeiro de Baker,et al.  Affective and behavioral predictors of novice programmer achievement , 2009, ITiCSE.

[2]  Frederick W. B. Li,et al.  Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior , 2013, 2013 IEEE 13th International Conference on Advanced Learning Technologies.

[3]  Frederick W. B. Li,et al.  No tests required: comparing traditional and dynamic predictors of programming success , 2014, SIGCSE.

[4]  Michael Kölling,et al.  Objects First with Java - A Practical Introduction Using BlueJ (5th Edition) , 2002 .

[5]  Michael Kölling,et al.  The BlueJ System and its Pedagogy , 2003, Comput. Sci. Educ..

[6]  Neil Brown,et al.  Investigating novice programming mistakes: educator beliefs vs. student data , 2014, ICER '14.

[7]  Andreas Stefik,et al.  An Empirical Investigation into Programming Language Syntax , 2013, TOCE.

[8]  Thomas R. G. Green,et al.  Cognitive dimensions of notations , 1990 .

[9]  Neil Brown,et al.  Blackbox: a large scale repository of novice programmers' activity , 2014, SIGCSE.

[10]  Matthew C. Jadud,et al.  Methods and tools for exploring novice compilation behaviour , 2006, ICER '06.

[11]  Ruud H. Koning,et al.  Statistical thinking in sports , 2007 .

[12]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[13]  K. Pearson.  Contributions to the Mathematical Theory of Evolution , 1894 .

[14]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[15]  Gerard H. Kuper,et al.  Statistical Thinking in Sports , 2007 .

[16]  K. Pearson.  Contributions to the Mathematical Theory of Evolution. II. Skew Variation in Homogeneous Material , 1895 .

[17]  Neil Brown,et al.  37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data , 2015, SIGCSE.

[18]  Ma. Mercedes T. Rodrigo,et al.  An Analysis of Java Programming Behaviors, Affect, Perceptions, and Syntax Errors among Low-Achieving, Average, and High-Achieving Novice Programmers , 2013 .

[19]  Ma. Mercedes T. Rodrigo,et al.  Predicting at-risk novice Java programmers through the analysis of online protocols , 2011, ICER.

[20]  Stephen H. Edwards,et al.  Web-CAT: automatically grading programming assignments , 2008, ITiCSE.