The impact of Software Testing education on code reliability: An empirical assessment

Abstract Software Testing (ST) is an indispensable part of software development. Proper testing education is thus of paramount importance. Indeed, the mere exposition to ST knowledge might have an impact on programming skills. In particular, it can encourage the production of more correct - and thus reliable - code. Although this is intuitive, to the best of our knowledge, there are no studies about such effects. Concerned with this, we have conducted two investigations related to ST education: (1) a large experiment with students to evaluate the possible impact of ST knowledge on the production of reliable code; and (2) a survey with professors that teach introductory programming courses to evaluate their level of ST knowledge. Our study involved 60 senior-level computer science students, 8 auxiliary functions with 92 test cases, a total of 248 implementations, and 53 professors of diverse subfields that completed our survey. The investigation with students shows that ST knowledge can improve code reliability in terms of correctness in as much as 20%, on average. On the other hand, the survey with professors reveals that, in general, university instructors tend to lack the same knowledge that would help students increase their programming skills toward more reliable code.

[1]  Sebastian G. Elbaum,et al.  Bug Hunt: Making Early Software Testing Lessons Engaging and Affordable , 2007, 29th International Conference on Software Engineering (ICSE'07).

[2]  David S. Janzen,et al.  Test-driven learning: intrinsic integration of testing into the CS/SE curriculum , 2006, SIGCSE '06.

[3]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[4]  Jeffrey C. Carver,et al.  A checklist for integrating student empirical studies with research and teaching goals , 2010, Empirical Software Engineering.

[5]  Mario Piattini,et al.  Evaluating performances of pair designing in industry , 2007, J. Syst. Softw..

[6]  Natalia Juristo Juzgado,et al.  Are Students Representatives of Professionals in Software Engineering Experiments? , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[7]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[8]  Eli M. Dow,et al.  The Emerging Role of Software Testing in Curricula , 2010, 2010 IEEE Transforming Engineering Education: Creating Interdisciplinary Skills for Complex Global Environments.

[9]  Laurie A. Williams,et al.  What agile teams think of agile principles , 2012, Commun. ACM.

[10]  Fabiano Cutigi Ferrari,et al.  Development of auxiliary functions: Should you be agile? An empirical assessment of pair programming and test-first programming , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[11]  Vinton G. Cerf.  Responsible programming , 2014, CACM.

[12]  Mary Jean Harrold,et al.  Testing: a roadmap , 2000, ICSE '00.

[13]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[14]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[15]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[16]  Edward L. Jones,et al.  Integrating testing into the curriculum — arsenic in small doses , 2001, SIGCSE '01.

[17]  Michael Kölling,et al.  Introducing unit testing with BlueJ , 2003 .

[18]  A. Jefferson Offutt,et al.  Using abstraction and Web applications to teach criteria-based test design , 2011, 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T).

[19]  Lionel C. Briand,et al.  Empirical studies of software testing techniques: challenges, practical strategies, and future research , 2004, SOEN.

[20]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[21]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[22]  Fabiano Cutigi Ferrari,et al.  Experience report: Can software testing education lead to more reliable code? , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[23]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[24]  W. Eric Wong,et al.  Improving the State of Undergraduate Software Testing Education , 2012 .

[25]  Lech Madeyski,et al.  Judy - a mutation testing tool for Java , 2010, IET Softw..

[26]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[27]  Tibor Gyimóthy,et al.  Empirical validation of object-oriented metrics on open source software for fault prediction , 2005, IEEE Transactions on Software Engineering.

[28]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[29]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[30]  Jeffrey C. Carver,et al.  Evaluating the testing ability of senior-level computer science students , 2011, 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T).

[31]  Tariq M. King,et al.  Integrating Testing into Software Engineering Courses Supported by a Collaborative Learning Environment , 2014, TOCE.