Adaptive Immediate Feedback Can Improve Novice Programming Engagement and Intention to Persist in Computer Science

Prior work suggests that novice programmers are greatly impacted by the feedback provided by their programming environments. While some research has examined the impact of feedback on student learning in programming, there is no work (to our knowledge) that examines the impact of adaptive immediate feedback within programming environments on students' desire to persist in computer science (CS). In this paper, we integrate an adaptive immediate feedback (AIF) system into a block-based programming environment. Our AIF system is novel because it provides personalized positive and corrective feedback to students in real time as they work. In a controlled pilot study with novice high-school programmers, we show that our AIF system significantly increased students' intentions to persist in CS, and that students using AIF had greater engagement (as measured by their lower idle time) compared to students in the control condition. Further, we found evidence that the AIF system may improve student learning, as measured by student performance in a subsequent task without AIF. In interviews, students found the system fun and helpful, and reported feeling more focused and engaged. We hope this paper spurs more research on adaptive immediate feedback and the impact of programming environments on students' intentions to persist in CS.

[1]  Moira Maguire,et al.  Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. , 2017 .

[2]  Thomas W. Price,et al.  Hint Generation Under Uncertainty: The Effect of Hint Quality on Help-Seeking Behavior , 2017, AIED.

[3]  Thomas Andre,et al.  Delayed retention effect: Attention, perseveration, or both? , 1989 .

[4]  Susanne Narciss,et al.  How to design informative tutoring feedback for multi-media learning , 2004 .

[5]  John R. Anderson,et al.  Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes , 2001, CHI.

[6]  Nicholas Lytle,et al.  Crescendo: Engaging Students to Self-Paced Programming Practices , 2020, SIGCSE.

[7]  Antonija Mitrovic,et al.  Assessing the Impact of Positive Feedback in Constraint-Based Tutors , 2008, Intelligent Tutoring Systems.

[8]  Thomas W. Price,et al.  Defining Tinkering Behavior in Open-ended Block-based Programming Assignments , 2019, SIGCSE.

[9]  BiggersMaureen,et al.  Student perceptions of computer science , 2008 .

[10]  Vincent Aleven,et al.  Help Helps, But Only So Much: Research on Help Seeking with Intelligent Tutoring Systems , 2016, International Journal of Artificial Intelligence in Education.

[11]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[12]  V. Shute.  Focus on Formative Feedback , 2008 .

[13]  Shelley J. Correll,et al.  Constraints into Preferences: Gender, Status, and Emerging Career Aspirations , 2004 .

[14]  Brett A. Becker,et al.  Effective compiler error message enhancement for novice programming students , 2016, Comput. Sci. Educ..

[15]  Colleen M. Lewis,et al.  Deciding to major in computer science: a grounded theory of students' self-assessment of ability , 2011, ICER.

[16]  David Hovemeyer,et al.  CloudCoder: a web-based programming exercise system , 2013 .

[17]  Lecia Jane Barker,et al.  Exploring factors that influence computer science introductory course students to persist in the major , 2009, SIGCSE '09.

[18]  David Weintrop,et al.  Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs , 2015, ICER.

[19]  Antonija Mitrovic,et al.  The effect of positive feedback in a constraint-based intelligent tutoring system , 2013, Comput. Educ..

[20]  John T. Richards,et al.  Designing Engaging Learning Experiences in Programming , 2017, CSEDU 2017.

[21]  Davide Fossati,et al.  Beyond the code-and-count analysis of tutoring dialogues , 2007, AIED.

[22]  Joseph Jay Williams,et al.  The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment , 2019, ITiCSE.

[23]  Joshua D. Angrist,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2008 .

[24]  R. Mayer,et al.  Personalized messages that promote science learning in virtual environments , 2004 .

[25]  NordquistPete.  Providing accurate and timely feedback by automatically grading student programming labs , 2007 .

[26]  Ottar Hellevik,et al.  Linear versus logistic regression when the dependent variable is a dichotomy , 2009 .

[27]  C. Stein,et al.  Structural equation modeling. , 2012, Methods in molecular biology.

[28]  Rui Zhi.  Reducing the State Space of Programming Problems through Data-Driven Feature Detection , 2018 .

[29]  Thomas W. Price,et al.  iSnap: Towards Intelligent Tutoring in Novice Programming Environments , 2017, SIGCSE.

[30]  Shravan Vasishth,et al.  An Introduction to Linear Mixed Models , 2011 .

[31]  M. Thurlings,et al.  Understanding feedback: A learning theory perspective , 2013 .

[32]  Austin Cory Bart,et al.  Misconception-Driven Feedback: Results from an Experimental Study , 2018, ICER.

[33]  Susan J. Ashford,et al.  Reflections on the Looking Glass: A Review of Research on Feedback-Seeking Behavior in Organizations , 2003 .

[34]  Thomas W. Price,et al.  Factors Influencing Students' Help-Seeking Behavior while Programming with Human and Computer Tutors , 2017, ICER.

[35]  Daantje Derks,et al.  Emoticons and Online Message Interpretation , 2008 .

[36]  Kristy Elizabeth Boyer,et al.  Learner Characteristics and Feedback in Tutorial Dialogue , 2008 .

[37]  Sandra Katz,et al.  Gender, achievement, and persistence in an undergraduate computer science program , 2006, DATB.

[38]  Monica A. Riordan.  Emojis as Tools for Emotion Work: Communicating Affect in Text Messages , 2017 .

[39]  M. Lepper,et al.  Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. , 1993 .

[40]  Ayelet Fishbach,et al.  How feedback infl uences persistence, disengagement, and change in goal pursuit , 2012 .

[41]  Amy J. Ko,et al.  Personifying programming tool feedback improves novice programmers' learning , 2011, ICER.

[42]  Joseph Jay Williams,et al.  An Evaluation of the Impact of Automated Programming Hints on Performance and Learning , 2019, ICER.

[43]  James K. McAfee,et al.  Providing Performance Feedback to Teachers: A Review , 2004 .

[44]  Thomas W. Price,et al.  Comparing Textual and Block Interfaces in a Novice Programming Environment , 2015, ICER.

[45]  Joe Karaganis,et al.  The rise of the robo notice , 2015, Commun. ACM.

[46]  Toni M. Kempler,et al.  The Cambridge Handbook of the Learning Sciences: Motivation and Cognitive Engagement in Learning Environments , 2005 .

[47]  Gary Lewandowski,et al.  Debugging: the good, the bad, and the quirky -- a qualitative analysis of novices' strategies , 2008, SIGCSE '08.

[48]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[49]  Davide Fossati,et al.  Data D riven A utomatic F eedback G eneration in the iList I ntelligent T utoring S ystem , 2014 .

[50]  Graham Holden,et al.  A role for technology in enhancing students’ engagement with feedback , 2012 .

[51]  V. Shute.  Focus on Formative Feedback , 2007 .

[52]  Stephen H. Edwards,et al.  CodeWorkout: Short Programming Exercises with Built-in Data Collection , 2017, ITiCSE.

[53]  Peter J. Denning,et al.  A technician shortage , 2015, Commun. ACM.

[54]  Samiha Marwan,et al.  Unproductive Help-seeking in Programming: What it is and How to Address it , 2020, ITiCSE.

[55]  D. Nicol,et al.  Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice , 2006 .

[56]  Tony Jenkins,et al.  The motivation of students of programming , 2001, ITiCSE '01.

[57]  Tuba Yilmaz,et al.  Student perceptions of computer science: a retention study comparing graduating seniors with cs leavers , 2008, SIGCSE '08.

[58]  Austin Cory Bart,et al.  Authoring feedback for novice programmers in a block-based language , 2017, 2017 IEEE Blocks and Beyond Workshop (B&B).

[59]  Kenneth R. Koedinger,et al.  Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor , 2015, International Journal of Artificial Intelligence in Education.

[60]  S. Ashford,et al.  Feedback-Seeking in Individual Adaptation: A Resource Perspective , 1986 .

[61]  Brian Rubineau,et al.  Professional Role Confidence and Gendered Persistence in Engineering , 2011 .

[62]  Pete Nordquist.  Providing Accurate And Timely Feedback By Automatically Grading Student Programming Labs , 2007, FECS.

[63]  Davide Fossati,et al.  Towards explaining effective tutorial dialogues , 2009 .

[64]  Brett A. Becker,et al.  The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test , 2018, SIGCSE.