Self-evaluation Interventions: Impact on Self-efficacy and Performance in Introductory Programming

Research has repeatedly shown self-efficacy to be associated with course outcomes in CS and across other fields. CS education research has documented this and has developed CS-specific self-efficacy measurement instruments, but to date there have been only a few studies examining interventions intended to improve students’ self-efficacy in CS, and several types of self-efficacy interventions suggested by previous research remain to be tested in CS. This study attempts to address this lack of research by reporting on the results of a trial intervention intended to improve students’ self-efficacy in an introductory programming course. Students were recruited to complete a self-evaluation task, which previous research has suggested could have a beneficial impact on self-efficacy, which should in turn have a beneficial impact on course performance. Participating students’ course outcomes and self-efficacy were compared with those of the students who did not complete the self-evaluation task, using propensity score weighting adjustments to control for differences between the groups on entering characteristics and prior values of self-efficacy and course outcomes. We found that, whereas there was only marginal evidence for the self-evaluation intervention having a direct effect on self-efficacy, students who completed the self-evaluation task had significantly higher project scores during the weeks they were asked to complete it, compared to the students who did not participate. These findings suggest that there are potential benefits to incorporating self-evaluation tasks into introductory CS courses, although perhaps not by virtue of directly influencing self-efficacy.

[1]  A. Bandura Self-Efficacy: The Exercise of Control , 1997, Journal of Cognitive Psychotherapy.

[2]  Jyoti Bhardwaj,et al.  In search of self-efficacy: development of a new instrument for first year Computer Science students , 2017, Comput. Sci. Educ..

[3]  J. Strijbos,et al.  Scaffolding Self-Regulated Learning Through Self-Assessment and Peer Assessment: Guidelines for Classroom Implementation , 2016 .

[4]  A. Azzouz 2011 , 2020, City.

[5]  Jan Vahrenhold,et al.  An instrument to assess self-efficacy in introductory algorithms courses , 2018, Inroads.

[6]  Keith Quille,et al.  CS1: how will they do? How can we help? A decade of research and practice , 2019, Comput. Sci. Educ..

[7]  Jens Bennedsen,et al.  Failure rates in introductory programming , 2007, SGCS.

[8]  Beth Simon,et al.  Experiencing programming assignments in CS1: the emotional toll , 2010, ICER '10.

[9]  J. D. Runyan,et al.  Using experience sampling to examine links between compassion, eudaimonia, and pro-social behavior. , 2018, Journal of personality.

[10]  Stef van Buuren,et al.  Flexible Imputation of Missing Data , 2012 .

[11]  Guido Alessandri,et al.  Academic Achievement: The Unique Contribution of Self-Efficacy Beliefs in Self-Regulated Learning beyond Intelligence, Personality Traits, and Self-Esteem. , 2013 .

[12]  Susan Wiedenbeck,et al.  Development and Validation of Scores on a Computer Programming Self-Efficacy Scale and Group Analyses of Novice Programmer Self-Efficacy , 1998 .

[13]  Frank M. Pajares,et al.  Sources of science self‐efficacy beliefs of middle school students , 2006 .

[14]  蕭瓊瑞撰述,et al.  2009 , 2019, The Winning Cars of the Indianapolis 500.

[15]  Karen Adams * Modelling success: enhancing international postgraduate research students' self‐efficacy for research seminar presentations , 2004 .

[16]  Daniel Zingaro Peer instruction contributes to self-efficacy in CS1 , 2014, SIGCSE '14.

[17]  K. Imai,et al.  Covariate balancing propensity score , 2014 .

[18]  Jan Vahrenhold,et al.  Revisiting Self-Efficacy in Introductory Programming , 2020, ICER.

[19]  Elizabeth A Stuart,et al.  Assessing the sensitivity of methods for estimating principal causal effects , 2015, Statistical methods in medical research.

[20]  Ernesto Panadero,et al.  To rubric or not to rubric? The effects of self-assessment on self-regulation, performance and self-efficacy , 2014 .

[21]  Barry J. Zimmerman,et al.  Becoming a Self-Regulated Writer: A Social Cognitive Perspective , 1997 .

[22]  D. McCaffrey,et al.  Propensity score estimation with boosted regression for evaluating causal effects in observational studies. , 2004, Psychological methods.

[23]  Eleanor O'Rourke,et al.  How Do Students Talk About Intelligence?: An Investigation of Motivation, Self-efficacy, and Mindsets in Computer Science , 2019, ICER.

[24]  How Many Imputations Do You Need? A Two-stage Calculation Using a Quadratic Rule , 2020 .

[25]  H. Cooper,et al.  The Relation Between Self-Beliefs and Academic Achievement: A Meta-Analytic Review , 2004 .

[26]  Karen Adams,et al.  Modelling success: enhancing international postgraduate research students' self‐efficacy for research seminar presentations , 2004 .

[27]  Jo Boaler,et al.  Ability and Mathematics: The Mindset Revolution that Is Reshaping Education. , 2013 .

[28]  V. Gecas The Social Psychology of Self-Efficacy , 1989 .

[29]  Paul T. von Hippel,et al.  How Many Imputations Do You Need? A Two-stage Calculation Using a Quadratic Rule , 2020 .

[30]  Peggy A. Ertmer,et al.  Chapter 19 – Self-Regulation and Academic Learning: Self-Efficacy Enhancing Interventions , 2000 .

[31]  Marian M. Larisey,et al.  Student Self-Assessment , 1994 .

[32]  Sarah Milne Self-efficacy in changing societies , 1999, Journal of health psychology.

[33]  M. Csíkszentmihályi,et al.  Experience Sampling Method: Measuring the Quality of Everyday Life , 2006 .

[34]  P. Gore Academic Self-Efficacy as a Predictor of College Outcomes: Two Incremental Validity Studies , 2006 .

[35]  H. Seway,et al.  1964 , 2019, Lie on your wounds.

[36]  Christopher D. Hundhausen,et al.  With a Little Help From My Friends: An Empirical Study of the Interplay of Students' Social Activities, Programming Activities, and Course Success , 2016, ICER.

[37]  Christopher D. Hundhausen,et al.  Talking about code: Integrating pedagogical code reviews into early computing courses , 2013, TOCE.

[38]  S. Normand,et al.  Intent-to-Treat vs. Non-Intent-to-Treat Analyses under Treatment Non-Adherence in Mental Health Randomized Trials. , 2008, Psychiatric annals.

[39]  Frederick W. B. Li,et al.  Failure rates in introductory programming revisited , 2014, ITiCSE '14.

[40]  Joshua Levi Weese,et al.  Increasing Student Self-Efficacy in Computational Thinking via STEM Outreach Programs , 2018, SIGCSE.

[41]  Beth Simon,et al.  CS majors' self-efficacy perceptions in CS1: results in light of social cognitive theory , 2011, ICER.

[42]  Wen-Hsiung Wu,et al.  An Investigation of Self-Regulatory Mechanisms in Learning to Program Visual Basic , 2013 .

[43]  Jesús Alonso-Tapia,et al.  Rubrics vs. self-assessment scripts: effects on first year university students’ self-regulation and performance / Rúbricas y guiones de autoevaluación: efectos sobre la autorregulación y el rendimiento de estudiantes universitarios de primer año , 2014 .

[44]  Ian R White,et al.  Should multiple imputation be the method of choice for handling missing data in randomized trials? , 2016, Statistical methods in medical research.

[45]  L. Hedges,et al.  Intraclass Correlation Values for Planning Group-Randomized Trials in Education , 2007 .

[46]  Ernesto Panadero,et al.  Rubrics and Self-Assessment Scripts Effects on Self-Regulation, Learning and Self-Efficacy in Secondary Education. , 2012 .

[47]  A. Bandura Self-efficacy: toward a unifying theory of behavioral change. , 1977, Psychological review.

[48]  Donald B. Rubin,et al.  Matching With Multiple Control Groups With Adjustment for Group Differences , 2008 .

[49]  J. Durlak How to select, calculate, and interpret effect sizes. , 2009, Journal of pediatric psychology.

[50]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[51]  P. Pintrich,et al.  Motivational and self-regulated learning components of classroom academic performance. , 1990 .

[52]  Greg Ridgeway,et al.  Toolkit for Weighting and Analysis of Nonequivalent Groups , 2014 .

[53]  Dan Cernusca,et al.  Can undergraduates learn programming with a “Virtual Professor”? Findings from a pilot implementation of a blended instructional strategy , 2013 .

[54]  J. Alonso-Tapia,et al.  Effects of self-assessment scripts on self-regulation and learning , 2010 .

[55]  Beth Simon,et al.  My program is ok – am I? Computing freshmen's experiences of doing programming assignments , 2012, Comput. Sci. Educ..

[56]  Lena Osterhagen,et al.  Multiple Imputation For Nonresponse In Surveys , 2016 .

[57]  Marilyn E. Gist,et al.  Self-Efficacy: A Theoretical Analysis of Its Determinants and Malleability , 1992 .

[58]  John L.P. Thompson,et al.  Missing data , 2004 .

[59]  Richard E. Lucas,et al.  Assessing the validity of single-item life satisfaction measures: results from three large samples , 2014, Quality of Life Research.

[60]  Anders Jonsson,et al.  Effects of self-assessment on self-regulated learning and self-efficacy: Four meta-analyses , 2017 .

[61]  Peter Z. Schochet,et al.  Report : Estimation and Identification of the Complier Average Causal Effect Parameter in Education RCTs , 2009 .

[62]  S. Hewitt,et al.  2007 , 2018, Los 25 años de la OMC: Una retrospectiva fotográfica.

[63]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[64]  D. Schunk Self-Efficacy and Education and Instruction , 1995 .

[65]  U. Nett,et al.  Adding previous experiences to the person-situation debate of achievement emotions , 2019, Contemporary Educational Psychology.

[66]  M. Credé,et al.  A meta-analytic review of the Motivated Strategies for Learning Questionnaire , 2011 .

[67]  Colleen M. Lewis,et al.  Competitive Enrollment Policies in Computing Departments Negatively Predict First-Year Students' Sense of Belonging, Self-Efficacy, and Perception of Department , 2020, SIGCSE.

[68]  S. Andrew,et al.  Self-efficacy as a predictor of academic performance in science. , 1998, Journal of advanced nursing.

[69]  Susan Wiedenbeck,et al.  Self-efficacy and mental models in learning to program , 2004, ITiCSE '04.

[70]  F. Vanclay,et al.  Principles for ethical research involving humans: ethical professional practice in impact assessment Part I , 2013 .

[71]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2000 .

[72]  Kristy J. Lauver,et al.  Do psychosocial and study skill factors predict college outcomes? A meta-analysis. , 2004, Psychological bulletin.

[73]  Romain Martin,et al.  “My Questionnaire is Too Long!” The assessments of motivational-affective constructs with three-item and single-item measures , 2014 .

[74]  Steven D. Brown,et al.  Relation of self-efficacy beliefs to academic outcomes: A meta-analytic investigation. , 1991 .

[75]  T. Williams,et al.  Self-Efficacy and Performance in Mathematics: Reciprocal Determinism in 33 Nations. , 2010 .

[76]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[77]  AERA Code of Ethics: American Educational Research Association Approved by the AERA Council February 2011 , 2011 .

[78]  Lane F Burgette,et al.  A tutorial on propensity score estimation for multiple treatments using generalized boosted models , 2013, Statistics in medicine.

[79]  C. Drake Effects of misspecification of the propensity score on estimators of treatment effect , 1993 .

[80]  J. Robins,et al.  Marginal Structural Models and Causal Inference in Epidemiology , 2000, Epidemiology.

[81]  A. Bandura,et al.  Self-Efficacy: The Exercise of Control , 1999, Journal of Cognitive Psychotherapy.

[82]  Gavin T. L. Brown,et al.  The Future of Student Self-Assessment: a Review of Known Unknowns and Potential Directions , 2016 .

[83]  Neena Thota,et al.  Pass Rates in Introductory Programming and in other STEM Disciplines , 2019, ITiCSE-WGR.

[84]  Woogul Lee,et al.  Testing interest and self-efficacy as predictors of academic self-regulation and achievement , 2014 .

[85]  Frederick W. B. Li,et al.  No tests required: comparing traditional and dynamic predictors of programming success , 2014, SIGCSE.

[86]  Ernesto Panadero,et al.  Rubrics vs. self-assessment scripts effect on self-regulation, performance and self-efficacy in pre-service teachers , 2013 .

[87]  Aman Yadav,et al.  Students' Emotional Reactions to Programming Projects in Introduction to Programming: Measurement Approach and Influence on Learning Outcomes , 2017, ICER.

[88]  Jeffrey M. Wooldridge,et al.  Inverse probability weighted M-estimators for sample selection, attrition, and stratification , 2002 .

[89]  Frank M. Pajares,et al.  Role of self-efficacy and self-concept beliefs in mathematical problem solving: A path analysis. , 1994 .

[90]  David P French,et al.  Which Behaviour Change Techniques Are Most Effective at Increasing Older Adults’ Self-Efficacy and Physical Activity Behaviour? A Systematic Review , 2014, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[91]  Aman Yadav,et al.  Learning to Program: Gender Differences and Interactive Effects of Students' Motivation, Goals, and Self-Efficacy on Performance , 2016, ICER.

[92]  J. Broadbent,et al.  The influence of academic self-efficacy on academic performance: A systematic review , 2016 .

[93]  A. Bandura Social Foundations of Thought and Action , 1986 .

[94]  Lisa Zhang,et al.  Impact of Open-Ended Assignments on Student Self-Efficacy in CS1 , 2019, CompEd.

[95]  William R. Shadish,et al.  A Primer on Propensity Score Analysis , 2010 .

[96]  Alex Lishinski,et al.  Cognitive, Affective, and Dispositional Components of Learning Programming , 2016, ICER.

[97]  Joel M. Hektner,et al.  Experience sampling method , 2007 .

[98]  Shangfeng Hu,et al.  Computer science skills across China, India, Russia, and the United States , 2019, Proceedings of the National Academy of Sciences.

[99]  Sylvia Beyer,et al.  Why are women underrepresented in Computer Science? Gender differences in stereotypes, self-efficacy, values, and interests and predictors of future CS course-taking and grades , 2014, Comput. Sci. Educ..

[100]  S. Cassidy Resilience Building in Students: The Role of Academic Self-Efficacy , 2015, Front. Psychol..

[101]  Susan Wiedenbeck,et al.  Factors affecting the success of non-majors in learning to program , 2005, ICER '05.

[102]  Åge Diseth,et al.  Self-beliefs among students: Grade level and gender differences in self-esteem, self-efficacy and implicit theories of intelligence , 2014 .

[103]  P. Pintrich,et al.  Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq) , 1993 .

[104]  Patrick Royston,et al.  Multiple Imputation of Missing Values: Update of Ice , 2005 .

[105]  A. Bandura Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[106]  R. Lent,et al.  Sociocognitive mechanisms of personal agency in career development: Pantheoretical prospects. , 1994 .

[107]  Frank Pajares,et al.  Influence of self-efficacy on elementary students' writing , 1997 .

[108]  F. Lyddy,et al.  Self-efficacy and short-term memory capacity as predictors of proportional reasoning , 2013 .