Design Patterns in Software Maintenance: An Experiment Replication at UPM - Experiences with the RESER'11 Joint Replication Project

Replication of software engineering experiments is crucial for dealing with validity threats to experiments in this area. Even though the empirical software engineering community is aware of the importance of replication, the replication rate is still very low. The RESER'11 Joint Replication Project aims to tackle this problem by simultaneously running a series of several replications of the same experiment. In this article, we report the results of the replication run at the Universidad Politécnica de Madrid. Our results are inconsistent with the original experiment. However, we have identified possible causes for them. We also discuss our experiences (in terms of pros and cons) during the replication.

[1]  Walter F. Tichy,et al.  A Controlled Experiment Comparing the Maintainability of Programs Designed with and without Design Patterns—A Replication in a Real Programming Environment , 2004, Empirical Software Engineering.

[2]  Jeffrey C. Carver.  Towards Reporting Guidelines for Experimental Replications: A Proposal , 2010 .

[3]  Erik Kamsties,et al.  An Empirical Evaluation of Three Defect-Detection Techniques , 1995, ESEC.

[4]  Jeffrey C. Carver,et al.  A Framework for Software Engineering Experimental Replications , 2008, 13th IEEE International Conference on Engineering of Complex Computer Systems (iceccs 2008).

[5]  Victor R. Basili,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987, IEEE Transactions on Software Engineering.

[6]  Natalia Juristo Juzgado,et al.  Replications types in experimental disciplines , 2010, ESEM '10.

[7]  Oscar Dieste,et al.  Hidden Evidence Behind Useless Replications , 2010 .

[8]  James Miller,et al.  An empirical evaluation of defect detection techniques , 1997, Inf. Softw. Technol..

[9]  Natalia Juristo Juzgado,et al.  Using differences among replications of software engineering experiments to gain knowledge , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[10]  Walter F. Tichy,et al.  A Controlled Experiment in Maintenance Comparing Design Patterns to Simpler Solutions , 2001, IEEE Trans. Software Eng..

[11]  D. Ridley,et al.  Cape Town, South Africa , 1986, Journal of clinical ultrasound : JCU.

[12]  Natalia Juristo Juzgado,et al.  The role of non-exact replications in software engineering experiments , 2011, Empirical Software Engineering.

[13]  Natalia Juristo Juzgado,et al.  Analysis of the influence of communication between researchers on experiment replication , 2006, ISESE '06.

[14]  Jeffrey C. Carver,et al.  Knowledge-Sharing Issues in Experimental Software Engineering , 2004, Empirical Software Engineering.

[15]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..