Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts

This Innovative Practice Full Paper presents an approach of using software development artifacts to gauge student behavior and the effectiveness of changes to curriculum design. There is an ongoing need to adapt university courses to changing requirements and shifts in industry. As an educator it is therefore vital to have access to methods, with which to ascertain the effects of curriculum design changes. In this paper, we present our approach of analyzing software repositories in order to gauge student behavior during project work. We evaluate this approach in a case study of a university undergraduate software development course teaching agile development methodologies. Surveys revealed positive attitudes towards the course and the change of employed development methodology from Scrum to Kanban. However, surveys were not usable to ascertain the degree to which students had adapted their workflows and whether they had done so in accordance with course goals. Therefore, we analyzed students’ software repository data, which represents information that can be collected by educators to reveal insights into learning successes and detailed student behavior. We analyze the software repositories created during the last five courses, and evaluate differences in workflows between Kanban and Scrum usage.

[1]  Daniel Roos,et al.  The machine that changed the world : the story of lean production , 1991 .

[2]  G. Melnik,et al.  A cross-program investigation of students' perceptions of agile methods , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[3]  Pekka Abrahamsson,et al.  On the Impact of Kanban on Software Project Work: An Empirical Case Study Investigation , 2011, 2011 16th IEEE International Conference on Engineering of Complex Computer Systems.

[4]  Letha H. Etzkorn,et al.  Predicting students' grades in computer science courses based on complexity measures of teacher's lecture notes , 2009 .

[5]  Neil T. Heffernan,et al.  Informing Teachers Live about Student Learning: Reporting in Assistment System , 2005 .

[6]  WilsonBrenda Cantwell,et al.  Contributing to success in an introductory computer science course , 2001 .

[7]  Rebecca Brent,et al.  How to Improve Teaching Quality , 1999 .

[8]  Tutut Herawan,et al.  A Systematic Review on Educational Data Mining , 2017, IEEE Access.

[9]  Kenneth R. Koedinger,et al.  An Open Repository and analysis tools for fine-grained, longitudinal learner data , 2008, EDM.

[10]  Lior Rokach,et al.  Predicting Student Exam's Scores by Analyzing Social Network Data , 2012, AMT.

[11]  Michael J. Rees,et al.  A feasible user story tool for agile software development? , 2002, Ninth Asia-Pacific Software Engineering Conference, 2002..

[12]  Sebastián Ventura,et al.  Knowledge Discovery with Genetic Programming for Providing Feedback to Courseware Authors , 2004, User Modeling and User-Adapted Interaction.

[13]  Mark Warschauer,et al.  Predicting MOOC performance with Week 1 Behavior , 2014, EDM.

[14]  Qin Zhang,et al.  Practical automated process and product metric collection and analysis in a classroom setting: lessons learned from Hackystat-UH , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[15]  Mung Chiang,et al.  MOOC performance prediction via clickstream data and social learning networks , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[16]  Denise A. Bonebright.  40 years of storming: a historical review of Tuckman's model of small group development , 2010 .

[17]  Giuseppe Riccardi,et al.  Predicting students' final exam scores from their course activities , 2015, 2015 IEEE Frontiers in Education Conference (FIE).

[18]  Jens Bennedsen,et al.  Optimists have more fun, but do they learn better? On the influence of emotional and social factors on learning introductory computer science , 2008, Comput. Sci. Educ..

[19]  Keith J. Topping,et al.  Methodological quandaries in studying process and outcomes in peer assessment , 2010 .

[20]  Casper Lassenius,et al.  Do High and Low Performing Student Teams Use Scrum Differently in Capstone Projects? , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering Education and Training Track (ICSE-SEET).

[21]  Sebastián Ventura,et al.  Educational Data Mining: A Review of the State of the Art , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[22]  Iván Martínez-Ortiz,et al.  E-Learning standards and learning analytics. Can data collection be improved by using standard data models? , 2013, 2013 IEEE Global Engineering Education Conference (EDUCON).

[23]  Henrik Kniberg,et al.  Kanban and Scrum - Making the Most of Both , 2010 .

[24]  Christoph Matthies,et al.  Teaching Agile the Agile Way — Employing Self-Organizing Teams in a Uni- versity Software Engineering Course , 2016 .

[25]  Ozren Gamulin,et al.  Data mining in hybrid learning: Possibility to predict the final exam result , 2013, 2013 36th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[26]  Muhammad Ovais Ahmad,et al.  Kanban in software development: A systematic literature review , 2013, 2013 39th Euromicro Conference on Software Engineering and Advanced Applications.

[27]  Abram Hindle,et al.  Judging a Commit by Its Cover: Correlating Commit Message Entropy with Build Status on Travis-CI , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[28]  Bernd Brügge,et al.  Team Composition in Software Engineering Project Courses , 2018, 2018 IEEE/ACM International Workshop on Software Engineering Education for Millennials (SEEM).

[29]  Emad Shihab,et al.  Commit guru: analytics and risk prediction of software commits , 2015, ESEC/SIGSOFT FSE.

[30]  James Noble,et al.  Organizing self-organizing teams , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[31]  Ryan Polk.  Agile and Kanban in Coordination , 2011, 2011 AGILE Conference.

[32]  Mirko Perkusich,et al.  A systematic review on the use of Definition of Done on agile software development projects , 2017, EASE.

[33]  Christoph Matthies,et al.  Agile metrics for a university software engineering course , 2016, 2016 IEEE Frontiers in Education Conference (FIE).

[34]  Mira Kajko-Mattsson,et al.  Developer-driven big-bang process transition from Scrum to Kanban , 2011, ICSSP '11.

[35]  Forrest Shull,et al.  Are developers complying with the process: an XP study , 2010, ESEM '10.

[36]  Christoph Matthies,et al.  How Surveys, Tutors and Software Help to Assess Scrum Adoption in a Classroom Software Engineering Project , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[37]  Jan Renz,et al.  Taking Informed Action on Student Activity in MOOCs , 2017, L@S.

[38]  Viljan Mahnič.  From scrum to Kanban: introducing lean principles to a software engineering capstone course , 2015 .