Review of measurements used in computing education research and suggestions for increasing standardization

ABSTRACT Background and context: The variables that researchers measure and how they measure them are central in any area of research, including computing education. Which research questions can be asked and how they are answered depends on measurement. Objective: To summarize the commonly used variables and measurements in computing education and to compare them to best practices in measurement for human-subjects research. Method: Systematic literature review analyzing 197 papers published during 2013–2017 in computing education research venues. Findings: The review illuminates common practices related to: variables measured (including learner characteristics), measurements used, and type of data analysis. The paper lists standardized measurements that were used and highlights commonly used variables for which no standardized measures exist. Implications: The paper concludes with information about best practices currently being used in the community that should be continued, as well as pointing out practices that could be improved along with recommendations for how to begin to adopt those best practices.

[1]  Christopher Watson,et al.  A systematic review of approaches for teaching introductory programming and their influence on success , 2014, ICER '14.

[2]  Leen-Kiat Soh,et al.  Perceived Instrumentality and Career Aspirations in CS1 Courses: Change and Relationships with Achievement , 2016, ICER.

[3]  Linda Marshall,et al.  Exploration of Participation in Student Software Engineering Teams , 2016, ACM Trans. Comput. Educ..

[4]  Sally Hamouda,et al.  A basic recursion concept inventory , 2017, Comput. Sci. Educ..

[5]  Mark Guzdial,et al.  Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment , 2016, ICER.

[6]  Jyoti Bhardwaj,et al.  In search of self-efficacy: development of a new instrument for first year Computer Science students , 2017, Comput. Sci. Educ..

[7]  Jan Vahrenhold,et al.  Developing and validating test items for first-year computer science courses , 2014, Comput. Sci. Educ..

[8]  Michael C. Loui,et al.  A psychometric evaluation of the digital logic concept inventory , 2014, Comput. Sci. Educ..

[9]  H. Lips,et al.  Gender- and science-related attitudes as predictors of college students' academic choices , 1992 .

[10]  Huda Khayrallah,et al.  Mining data from the AP CS a exam: patterns, non-patterns, and replication failure , 2013, ICER '13.

[11]  G. Antes,et al.  Five Steps to Conducting a Systematic Review , 2003, Journal of the Royal Society of Medicine.

[12]  So Yoon Yoon,et al.  Psychometric properties of the Revised Purdue Spatial Visualization Tests: Visualization of Rotations (the Revised PSVT:R) , 2011 .

[13]  Alexander Repenning,et al.  Early validation of computational thinking pattern analysis , 2014, ITiCSE '14.

[14]  Jennifer M. Blaney,et al.  “But it doesn’t come naturally”: how effort expenditure shapes the benefit of growth mindset on women’s sense of intellectual belonging in computing , 2017, Comput. Sci. Educ..

[15]  David C. Webb,et al.  Scalable Game Design , 2015, ACM Trans. Comput. Educ..

[16]  Amy J. Ko,et al.  Computing Mentorship in a Software Boomtown: Relationships to Adolescent Interest and Beliefs , 2017, ICER.

[17]  Stephen Cooper,et al.  Spatial Skills Training in Introductory Computing , 2015, ICER.

[18]  Susan Wiedenbeck,et al.  Development and Validation of Scores on a Computer Programming Self-Efficacy Scale and Group Analyses of Novice Programmer Self-Efficacy , 1998 .

[19]  Frank LaBanca,et al.  Development of the Educational and Career Interest Scale in Science, Technology, and Mathematics for High School Students , 2013 .

[20]  Christopher D. Hundhausen,et al.  The Normalized Programming State Model: Predicting Student Performance in Computing Courses Based on Programming Behavior , 2015, ICER.

[21]  Mark Guzdial,et al.  Subgoals, Context, and Worked Examples in Learning Computing Problem Solving , 2015, ICER.

[22]  T. Gog,et al.  Development of an instrument for measuring different types of cognitive load , 2013, Behavior Research Methods.

[23]  S. Cross,et al.  The relational-interdependent self-construal and relationships. , 2000, Journal of personality and social psychology.

[24]  Saadeddine Shehab,et al.  Assessing collaborative computing: development of the Collaborative-Computing Observation Instrument (C-COI) , 2016, Comput. Sci. Educ..

[25]  Baruch B. Schwarz,et al.  Argumentation and Explanation in Conceptual Change: Indications From Protocol Analyses of Peer-to-Peer Dialog , 2009, Cogn. Sci..

[26]  Diana Franklin,et al.  K-8 Learning Trajectories Derived from Research Literature: Sequence, Repetition, Conditionals , 2017, ICER.

[27]  Aman Yadav,et al.  Methodological Rigor and Theoretical Foundations of CS Education Research , 2016, ICER.

[28]  Cynthia Bailey Lee,et al.  Can peer instruction be effective in upper-division computer science courses? , 2013, TOCE.

[29]  Jason Freeman,et al.  EarSketch: A STEAM-Based Approach for Underrepresented Populations in High School Computer Science Education , 2016, TOCE.

[30]  Monica McGill,et al.  Pre-College Computing Outreach Research: Towards Improving the Practice , 2017, SIGCSE.

[31]  Jan Vahrenhold,et al.  An instrument to assess self-efficacy in introductory algorithms courses , 2018, Inroads.

[32]  Monica McGill,et al.  Improving Research and Experience Reports of Pre-College Computing Activities: A Gap Analysis , 2018, SIGCSE.

[33]  Yifat Ben-David Kolikant,et al.  The effect of prior education on students’ competency in digital logic: the case of ultraorthodox Jewish students , 2017, Comput. Sci. Educ..

[34]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[35]  David Williamson Shaffer,et al.  Can We Rely on IRR? Testing the Assumptions of Inter-Rater Reliability , 2017, CSCL.

[36]  Ilenia Fronza,et al.  Teaching Computational Thinking Using Agile Software Engineering Methods , 2017, ACM Trans. Comput. Educ..

[37]  William M. K. Trochim,et al.  Research methods knowledge base , 2001 .

[38]  Allison Elliott Tew,et al.  Empirical validation and application of the computing attitudes survey , 2015, Comput. Sci. Educ..

[39]  Raymond Lister,et al.  Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance , 2015, ICER.

[40]  P. Pintrich,et al.  Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq) , 1993 .

[41]  Anthony V. Robins,et al.  The ongoing challenges of computer science education research , 2015, Comput. Sci. Educ..

[42]  Jürgen Börstler,et al.  Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies , 2015, ITiCSE-WGR.

[43]  Bruce W. Tuckman,et al.  The Development and Concurrent Validity of the Procrastination Scale , 1991 .

[44]  Michael F. Bunting,et al.  Measuring working memory is all fun and games: a four-dimensional spatial game predicts cognitive task performance. , 2014, Experimental psychology.

[45]  Stanislav Kurkovsky,et al.  Mobile game development: improving student engagement and motivation in introductory computing courses , 2013, Comput. Sci. Educ..

[46]  Michelle K. Smith,et al.  Why Peer Discussion Improves Student Performance on In-Class Concept Questions , 2009, Science.

[47]  Mark Guzdial,et al.  Developing a validated assessment of fundamental CS1 concepts , 2010, SIGCSE.

[48]  Janet Rountree,et al.  Interacting factors that predict success and failure in a CS1 course , 2004, ACM SIGCSE Bull..

[49]  Lisa S. Blackwell,et al.  Implicit theories of intelligence predict achievement across an adolescent transition: a longitudinal study and an intervention. , 2007, Child development.

[50]  Stephen H. Edwards,et al.  The Effects of Procrastination Interventions on Programming Project Success , 2015, ICER.

[51]  Ibrahim Çetin,et al.  Visualization: a tool for enhancing students’ concept images of basic object-oriented concepts , 2013, Comput. Sci. Educ..

[52]  Frank Fischer,et al.  What Do We Teach When We Teach the Learning Sciences? A Document Analysis of 75 Graduate Programs , 2018 .

[53]  John K. Bennett,et al.  Computer Science Principles: Impacting Student Motivation & Learning Within and Beyond the Classroom , 2016, ICER.

[54]  Brian Dorn,et al.  Aggregate Compilation Behavior: Findings and Implications from 27,698 Users , 2015, ICER.

[55]  Saadeddine Shehab,et al.  Describing Elementary Students' Interactions in K-5 Puzzle-based Computer Science Environments using the Collaborative Computing Observation Instrument (C-COI) , 2017, ICER.

[56]  Kurt Schneider,et al.  Media, Mood, and Meetings: Related to Project Success? , 2015, TOCE.

[57]  S. Srivastava,et al.  The Big Five Trait taxonomy: History, measurement, and theoretical perspectives. , 1999 .

[58]  R. Meredith Belbin,et al.  Team Roles at Work , 2022 .

[59]  Olle Bälter,et al.  Gender neutrality improved completion rate for all , 2016, Comput. Sci. Educ..

[60]  Jason Chen,et al.  Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers , 2016, ICER.

[61]  Briana B. Morrison.  Dual Modality Code Explanations for Novices: Unexpected Results , 2017, ICER.

[62]  Neil Brown,et al.  Investigating novice programming mistakes: educator beliefs vs. student data , 2014, ICER '14.

[63]  Yuning Xu,et al.  Principled Assessment of Student Learning in High School Computer Science , 2017, ICER.

[64]  R. Keith Sawyer,et al.  Foundations of the learning sciences , 2014 .

[65]  Janet Rountree,et al.  Interacting factors that predict success and failure in a CS1 course , 2004, ITiCSE-WGR '04.

[66]  Reinhard Pekrun,et al.  Development of mathematical competencies in adolescence : the PALMA longitudinal study , 2007 .

[67]  Sylvia Beyer,et al.  Why are women underrepresented in Computer Science? Gender differences in stereotypes, self-efficacy, values, and interests and predictors of future CS course-taking and grades , 2014, Comput. Sci. Educ..

[68]  R. Estrella,et al.  Stuck in the Shallow End Education , Race , and Computing , 2008 .

[69]  Amy J. Ko,et al.  Comprehension First: Evaluating a Novel Pedagogy and Tutoring System for Program Tracing in CS1 , 2017, ICER.

[70]  Christopher D. Hundhausen,et al.  Talking about code: Integrating pedagogical code reviews into early computing courses , 2013, TOCE.

[71]  Amber Settle,et al.  Towards a Common Framework for Evaluating Computing Outreach Activities , 2016, SIGCSE.

[72]  Alexander Repenning,et al.  The zones of proximal flow: guiding students through a space of computational thinking skills and challenges , 2013, ICER.

[73]  Ibrahim Çetin,et al.  Development of computer programming attitude scale for university students , 2015, Comput. Appl. Eng. Educ..

[74]  Jenefer Husman,et al.  Control, Motivation, Affect, and Strategic Self-Regulation in the College Classroom : A Multidimensional Phenomenon , 2008 .

[75]  George Ghinea,et al.  Measuring enrichment: the assembly and validation of an instrument to assess student self-beliefs in CS1 , 2014, ICER '14.

[76]  Quintin I. Cutts,et al.  Student experience in a student-centered peer instruction classroom , 2013, ICER.

[77]  I. B. Myers.  Manual: A Guide to the Development and Use of the Myers-Briggs Type Indicator , 1985 .

[78]  Raymond Lister,et al.  Predicting student success using fine grain clicker data , 2014, ICER '14.

[79]  Amy J. Ko,et al.  Comparing the Effectiveness of Online Learning Approaches on CS1 Learning Outcomes , 2015, ICER.

[80]  Kathi Fisler,et al.  The recurring rainfall problem , 2014, ICER '14.

[81]  Helene Gelderblom,et al.  A blended learning approach for teaching computer programming: design for large classes in Sub-Saharan Africa , 2014, Comput. Sci. Educ..

[82]  Jenefer Husman,et al.  Instrumentality, task value, and intrinsic motivation: Making sense of their independent interdependence , 2004 .

[83]  Leen-Kiat Soh,et al.  Exploring Changes in Computer Science Students' Implicit Theories of Intelligence Across the Semester , 2015, ICER.

[84]  Anthony S. Bryk,et al.  National Board for Education Sciences , 2014 .

[85]  M. Johns,et al.  A new method for measuring daytime sleepiness: the Epworth sleepiness scale. , 1991, Sleep.

[86]  Richard Catrambone,et al.  Using Learners' Self-Explanations of Subgoals to Guide Initial Problem Solving in App Inventor , 2017, ICER.

[87]  Rebecca T. Mercuri,et al.  Identifying and correcting Java programming errors for introductory computer science students , 2003, SIGCSE.

[88]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[89]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[90]  J. Biggs,et al.  The revised two-factor Study Process Questionnaire: R-SPQ-2F. , 2001, The British journal of educational psychology.

[91]  Maria Öhrstedt,et al.  Studieapproach, stress, studieresultat och förmåga att bedöma egen prestation , 2009 .

[92]  Giorgos Lepouras,et al.  How Do Different Cognitive Styles Affect Learning Programming? Insights from a Game-Based Approach in Greek Schools , 2016, ACM Trans. Comput. Educ..

[93]  Jim Rogers,et al.  Understanding Student Self-Reports of Academic Performance and Course-Taking Behavior , 2017 .

[94]  Mark Guzdial,et al.  Measuring cognitive load in introductory CS: adaptation of an instrument , 2014, ICER '14.