Exploring the Value of Student Self-Evaluation in Introductory Programming

Programming teachers have a strong need for easy-to-use instruments that provide reliable and pedagogically useful insights into student learning. Currently, no validated tools exist for rapidly assessing student understanding of basic programming knowledge. Concept inventories and the SCS1 questionnaire can offer great benefits; this article explores the additional value that may be gained from relatively simple self-evaluation metrics. We apply a lightweight self-evaluation instrument (SEI) in an introductory programming course and compare the results to existing performance measures, such as examination grades and the SCS1. We find that the SEI has a similar correlation with a program-writing examination as the SCS1 does, although both instruments correlate only moderately with the examination and each other. Furthermore, students are much more likely to voluntarily answer the lightweight SEI than SCS1. Overall, our results suggest that both the SEI and other instruments need to be greatly improved and outline future work towards that end.

[1]  Jan Vahrenhold,et al.  An instrument to assess self-efficacy in introductory algorithms courses , 2018, Inroads.

[2]  Allison Elliott Tew,et al.  A fresh look at novice programmers' performance and their teachers' expectations , 2013, ITiCSE -WGR '13.

[3]  Yingjun Cao,et al.  Developing Assessments to Determine Mastery of Programming Fundamentals , 2017, ITiCSE.

[4]  John Sweller,et al.  Extending Cognitive Load Theory to Incorporate Working Memory Resource Depletion: Evidence from the Spacing Effect , 2018 .

[5]  Kari Smolander,et al.  Student self-assessment in a programming course using bloom's revised taxonomy , 2010, ITiCSE '10.

[6]  Dastyni Loksa,et al.  A theory of instruction for introductory programming skills , 2019, Comput. Sci. Educ..

[7]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[8]  J. Ángel Velázquez-Iturbide,et al.  A Systematic Review of the Use of Bloom's Taxonomy in Computer Science Education , 2018, SIGCSE.

[9]  Marcos Romn-Gonzlez,et al.  Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test , 2017, Comput. Hum. Behav..

[10]  Raymond Lister,et al.  A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers , 2013, ACE '13.

[11]  Kathi Fisler,et al.  Harnessing the Wisdom of the Classes: Classsourcing and Machine Learning for Assessment Instrument Generation , 2019, SIGCSE.

[12]  B. Zimmerman.  Becoming a Self-Regulated Learner: An Overview , 2002 .

[13]  Christopher Watson,et al.  A systematic review of approaches for teaching introductory programming and their influence on success , 2014, ICER '14.

[14]  Lauren E. Margulieux,et al.  Review of measurements used in computing education research and suggestions for increasing standardization , 2019, Comput. Sci. Educ..

[15]  Jens F. Beckmann,et al.  Taming a Beast of Burden--On Some Issues with the Conceptualisation and Operationalisation of Cognitive Load. , 2010 .

[16]  E. Sanz,et al.  Reseña: Common european framework of reference for languages: Learning, teaching, assessment. Companion volume with new descriptors , 2018 .

[17]  David Weintrop,et al.  Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs , 2015, ICER.

[18]  Arto Hellas,et al.  Towards a Common Instrument for Measuring Prior Programming Knowledge , 2019, ITiCSE.

[19]  Susan Wiedenbeck,et al.  Factors affecting course outcomes in introductory programming , 2004, PPIG.

[20]  Min Li,et al.  An Item Response Theory Evaluation of a Language-Independent CS1 Knowledge Assessment , 2019, SIGCSE.

[21]  Michael C. Loui,et al.  Setting the Scope of Concept Inventories for Introductory Computing Subjects , 2010, TOCE.

[22]  Mark Guzdial,et al.  Developing a validated assessment of fundamental CS1 concepts , 2010, SIGCSE.

[23]  P. Miller,et al.  The Effect of Scoring Criteria Specificity on Peer and Self-assessment , 2003 .

[24]  Janet Carter,et al.  ITiCSE 2010 working group report motivating our top students , 2010, ITiCSE-WGR '10.

[25]  R. Hembree.  Correlates, Causes, Effects, and Treatment of Test Anxiety , 1988 .

[26]  George Ghinea,et al.  Measuring enrichment: the assembly and validation of an instrument to assess student self-beliefs in CS1 , 2014, ICER '14.

[27]  J. Sweller.  Element Interactivity and Intrinsic, Extraneous, and Germane Cognitive Load , 2010 .

[28]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[29]  Paul R. Pintrich,et al.  THE ROLE OF SELF-EFFICACY BELIEFS INSTUDENT ENGAGEMENT AND LEARNING INTHECLASSROOM , 2003 .

[30]  Cynthia Taylor,et al.  Leveraging open source principles for flexible concept inventory development , 2014, ITiCSE '14.

[31]  Dion Timmermann,et al.  Evidence-based re-design of an introductory course “programming in C” , 2016, 2016 IEEE Frontiers in Education Conference (FIE).

[32]  Brenda Cantwell Wilson,et al.  Contributing to success in an introductory computer science course: a study of twelve factors , 2001, SIGCSE '01.

[33]  Harriet G. Taylor,et al.  Exploration of the Relationship between Prior Computing Experience and Gender on Success in College Computer Science , 1994 .

[34]  A. Bandura.  Self-efficacy: toward a unifying theory of behavioral change. , 1977, Psychological review.

[35]  Angela Carbone,et al.  Performance and progression of first year ICT students , 2008, ACE '08.

[36]  D. Schunk.  Modeling and attributional effects on children's achievement: A self-efficacy analysis. , 1981 .

[37]  Sven Apel,et al.  Measuring programming experience , 2012, 2012 20th IEEE International Conference on Program Comprehension (ICPC).

[38]  Dianne Hagan,et al.  Does it help to have some programming experience before beginning a computing degree program? , 2000, ITiCSE '00.

[39]  Sue Sentance,et al.  Learning to use Functions: The Relationship Between Misconceptions and Self-Efficacy , 2019, SIGCSE.

[40]  Mark Guzdial,et al.  Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment , 2016, ICER.

[41]  Jonathan Gratch,et al.  (Re)Validating Cognitive Introductory Computing Instruments , 2019, SIGCSE.

[42]  Tony Clear,et al.  Report on the Eighth BRACElet workshop: BRACElet technical report 01/08 AUT university, Auckland , 2009 .

[43]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[44]  John A. Ross.  The Reliability, Validity, and Utility of Self-Assessment , 2006 .

[45]  Raymond Lister,et al.  Relationships between reading, tracing and writing skills in introductory programming , 2008, ICER '08.

[46]  إسلام يسري علي.  مراجعة كتاب الإطار المرجعي الأوروبي المشترك للغات: دراسة، تدريس، تقييم / The Common European Framework of Reference for Languages: Learning, Teaching, Assessment , 1970 .

[47]  Raymond Lister,et al.  Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance , 2015, ICER.

[48]  Raymond Lister,et al.  Toward a Developmental Epistemology of Computer Programming , 2016, WiPSCE.

[49]  Roy D. Pea,et al.  Assessing computational learning in K-12 , 2014, ITiCSE '14.

[50]  Cynthia Taylor,et al.  Computer science concept inventories: past and future , 2014, Comput. Sci. Educ..

[51]  Barbara A. Greene,et al.  Identification with academics, intrinsic/extrinsic motivation, and self-efficacy as predictors of cognitive engagement , 2006 .

[52]  Elliot Soloway,et al.  Learning to program = learning to construct mechanisms and explanations , 1986, CACM.

[53]  Amy J. Ko,et al.  Comparing the Effectiveness of Online Learning Approaches on CS1 Learning Outcomes , 2015, ICER.

[54]  Amy J. Ko,et al.  Comprehension First: Evaluating a Novel Pedagogy and Tutoring System for Program Tracing in CS1 , 2017, ICER.

[55]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[56]  Josh D. Tenenberg,et al.  Do computer science students know what they know?: a calibration study of data structure knowledge , 2005, ITiCSE '05.

[57]  Stephen Chi-fai Chan,et al.  On the implementation of self-assessment in an introductory programming course , 2010, SGCS.

[58]  Juha Sorva,et al.  Towards an Analysis of Program Complexity From a Cognitive Perspective , 2018, ICER.

[59]  T. Gog,et al.  Training self-assessment and task-selection skills: A cognitive approach to improving self-regulated learning , 2012 .

[60]  Donna Teague,et al.  Neo-Piagetian Theory and the novice programmer , 2015 .