No tests required: comparing traditional and dynamic predictors of programming success

Research over the past fifty years into predictors of programming performance has yielded little improvement in the identification of at-risk students. This is possibly because research to date is based upon using static tests, which fail to reflect changes in a student's learning progress over time. In this paper, the effectiveness of 38 traditional predictors of programming performance are compared to 12 new data-driven predictors, that are based upon analyzing directly logged data, describing the programming behavior of students. Whilst few strong correlations were found between the traditional predictors and performance, an abundance of strong significant correlations based upon programming behavior were found. A model based upon two of these metrics (Watwin score and percentage of lab time spent resolving errors) could explain 56.3% of the variance in coursework results. The implication of this study is that a student's programming behavior is one of the strongest indicators of their performance, and future work should continue to explore such predictors in different teaching contexts.

[1]  Matthew C. Jadud,et al.  Methods and tools for exploring novice compilation behaviour , 2006, ICER '06.

[2]  Brenda Cantwell Wilson,et al.  Contributing to success in an introductory computer science course: a study of twelve factors , 2001, SIGCSE '01.

[3]  P. Pintrich,et al.  Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq) , 1993 .

[4]  Lawrence S. Corman,et al.  Cognitive style, personality type, and learning ability as factors in predicting the success of the beginning programming student , 1986, SGCS.

[5]  Rynson W. H. Lau,et al.  A pedagogical interface for authoring adaptive e-learning courses , 2010, MTDL '10.

[6]  Wilfred W. F. Lau,et al.  Exploring the effects of gender and learning styles on computer programming performance: implications for programming pedagogy , 2009, Br. J. Educ. Technol..

[7]  Susan Wiedenbeck,et al.  Factors affecting the success of non-majors in learning to program , 2005, ICER '05.

[8]  Ronan G. Reilly,et al.  The Influence of Motivation and Comfort-Level on Learning to Program , 2005, PPIG.

[9]  Jens Bennedsen,et al.  Failure rates in introductory programming , 2007, SGCS.

[10]  Ronan G. Reilly,et al.  Programming: factors that influence success , 2005, SIGCSE '05.

[11]  Philip R. Ventura,et al.  Identifying predictors of success for an objects-first CS1 , 2005, Comput. Sci. Educ..

[12]  Garry L. White,et al.  An Empirical Investigation of the Relationship between Success in Mathematics and Visual Programming Courses , 2003, J. Inf. Syst. Educ..

[13]  Ryan Shaun Joazeiro de Baker,et al.  Affective and behavioral predictors of novice programmer achievement , 2009, ITiCSE.

[14]  Ronan G. Reilly,et al.  Examining the role of self-regulated learning on introductory programming performance , 2005, ICER '05.

[15]  Rynson W. H. Lau,et al.  Learning Programming Languages through Corrective Feedback and Concept Visualisation , 2011, ICWL.

[16]  Wilfred W. F. Lau,et al.  Modelling programming performance: Beyond the influence of learner characteristics , 2011, Comput. Educ..

[17]  A. T. Chamillard,et al.  Using learning style data in an introductory computer science course , 1999, SIGCSE '99.

[18]  Michael V. Stein,et al.  Mathematical preparation as a basis for success in CS-II , 2002 .

[19]  Michael Johnstone,et al.  The Significance of Learning Style with Respect to Achievement in First Year Programming Students , 2010, 2010 21st Australian Software Engineering Conference.

[20]  Christopher Watson,et al.  Game-based concept visualization for learning programming , 2011, MTDL '11.

[21]  Frederick W. B. Li,et al.  Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior , 2013, 2013 IEEE 13th International Conference on Advanced Learning Technologies.

[22]  Margaret Anne Pierce,et al.  Attributional style as a predictor of success in a first computer science course , 1993 .

[23]  Frederick W. B. Li,et al.  BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair , 2012, ICWL.

[24]  A. J. Biamonte.  Predicting success in programmer training , 1964, SIGCPR '64.

[25]  Pat Byrne,et al.  The effect of student attributes on success in programming , 2001, ITiCSE '01.