An industry experiment on the effects of test-driven development on external quality and productivity

Existing empirical studies on test-driven development (TDD) report different conclusions about its effects on quality and productivity. Very few of those studies are experiments conducted with software professionals in industry. We aim to analyse the effects of TDD on the external quality of the work done and the productivity of developers in an industrial setting. We conducted an experiment with 24 professionals from three different sites of a software organization. We chose a repeated-measures design, and asked subjects to implement TDD and incremental test last development (ITLD) in two simple tasks and a realistic application close to real-life complexity. To analyse our findings, we applied a repeated-measures general linear model procedure and a linear mixed effects procedure. We did not observe a statistical difference between the quality of the work done by subjects in both treatments. We observed that the subjects are more productive when they implement TDD on a simple task compared to ITLD, but the productivity drops significantly when applying TDD to a complex brownfield task. So, the task complexity significantly obscured the effect of TDD. Further evidence is necessary to conclude whether TDD is better or worse than ITLD in terms of external quality and productivity in an industrial setting. We found that experimental factors such as selection of tasks could dominate the findings in TDD studies.

[1]  Sami Kollanus,et al.  Test-Driven Development - Still a Promising Approach? , 2010, 2010 Seventh International Conference on the Quality of Information and Communications Technology.

[2]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[3]  Hae-Young Kim.  Statistical notes for clinical researchers: assessing normal distribution (2) using skewness and kurtosis , 2013, Restorative dentistry & endodontics.

[4]  S. R. Searle,et al.  Generalized, Linear, and Mixed Models , 2005 .

[5]  R. Coe,et al.  It's the Effect Size, Stupid What effect size is and why it is important , 2012 .

[6]  Roberto Latorre,et al.  A successful application of a Test-Driven Development strategy in the industrial environment , 2014, Empirical Software Engineering.

[7]  S. R. Searle,et al.  Generalized, Linear, and Mixed Models: McCulloch/Generalized, Linear, and Mixed Models , 2005 .

[8]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[9]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[10]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[11]  Mario Piattini,et al.  Evaluating advantages of test driven development: a controlled experiment with professionals , 2006, ISESE '06.

[12]  Natalia Juristo,et al.  Experiences Conducting Experiments in Industry: The ESEIL FiDiPro Project , 2016, 2016 IEEE/ACM 4th International Workshop on Conducting Empirical Studies in Industry (CESI).

[13]  Maria Siniaalto.  Test driven development: empirical body of evidence , 2007 .

[14]  Mauricio Finavaro Aniche,et al.  Most Common Mistakes in Test-Driven Development Practice: Results from an Online Survey with Developers , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[15]  Burak Turhan,et al.  A Replicated Experiment on the Effectiveness of Test-First Development , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[16]  Dietmar Pfahl,et al.  Reporting guidelines for controlled experiments in software engineering , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[17]  Roberto Latorre,et al.  Effects of Developer Experience on Learning and Applying Unit Test-Driven Development , 2014, IEEE Transactions on Software Engineering.

[18]  Marcelo Soares Pimenta,et al.  Besouro: A framework for exploring compliance rules in automatic TDD behavior assessment , 2015, Inf. Softw. Technol..

[19]  Laurie A. Williams,et al.  On the Sustained Use of a Test-Driven Development Practice at IBM , 2007, Agile 2007 (AGILE 2007).

[20]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[21]  Boby George,et al.  Analysis and Quantification of Test Driven Development Approach , 2002 .

[22]  Natalia Juristo Juzgado,et al.  Topic selection in industry experiments , 2014, CESI 2014.

[23]  V. Basili.  Software modeling and measurement: the Goal/Question/Metric paradigm , 1992 .

[24]  Daniel Sundmark,et al.  An Industrial Survey on Contemporary Aspects of Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[25]  Lech Madeyski,et al.  The Impact of Test-Driven Development on Software Development Productivity - An Empirical Study , 2007, EuroSPI.

[26]  Natalia Juristo Juzgado,et al.  Are Students Representatives of Professionals in Software Engineering Experiments? , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[27]  Laurie A. Williams,et al.  Assessing test-driven development at IBM , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[28]  Tore Dybå,et al.  Construction and Validation of an Instrument for Measuring Programming Skill , 2014, IEEE Transactions on Software Engineering.

[29]  Laurie A. Williams,et al.  Realizing quality improvement through test driven development: results and experiences of four industrial teams , 2008, Empirical Software Engineering.

[30]  Andrew Fish,et al.  Towards an Operationalization of the "Physics of Notations" for the Analysis of Visual Languages , 2013, MoDELS.

[31]  Julia Eichmann,et al.  Making Software - What Really Works, and Why We Believe It , 2011, Making Software.

[32]  Vojislav B. Misic,et al.  The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis , 2013, IEEE Transactions on Software Engineering.

[33]  Marco Torchiano,et al.  On the effectiveness of the test-first approach to programming , 2005, IEEE Transactions on Software Engineering.

[34]  Tore Dybå,et al.  A systematic review of effect size in software engineering experiments , 2007, Inf. Softw. Technol..

[35]  Boby George,et al.  An initial investigation of test driven development in industry , 2003, SAC '03.

[36]  Boby George,et al.  A structured experiment of test-driven development , 2004, Inf. Softw. Technol..

[37]  James Miller,et al.  A prototype empirical evaluation of test driven development , 2004 .

[38]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[39]  Natalia Juristo Juzgado,et al.  Towards an operationalization of test-driven development skills: An industrial empirical study , 2015, Inf. Softw. Technol..

[40]  Tony Gorschek,et al.  A method for evaluating rigor and industrial relevance of technology evaluations , 2011, Empirical Software Engineering.

[41]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[42]  Mojca Ciglaric,et al.  Impact of test-driven development on productivity, code and tests: A controlled experiment , 2011, Inf. Softw. Technol..

[43]  Laurie A. Williams,et al.  Test-driven development as a defect-reduction practice , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[44]  Daniel Sundmark,et al.  Factors Limiting Industrial Adoption of Test Driven Development: A Systematic Review , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[45]  Jouni Markkula,et al.  Survey on agile and lean usage in finnish software industry , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[46]  Jari Still.  Experiences in Applying Agile Software Development in F-Secure , 2007, PROFES.

[47]  Burak Turhan,et al.  Impact of process conformance on the effects of test-driven development , 2014, ESEM '14.

[48]  Kai Petersen,et al.  Considering rigor and relevance when evaluating test driven development: A systematic review , 2014, Inf. Softw. Technol..