Investigations about replication of empirical studies in software engineering: A systematic mapping study

ContextTwo recent mapping studies which were intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: empirical studies actually reporting replications (published between 1994 and 2012) and a second group of studies that are concerned with definitions, classifications, processes, guidelines, and other research topics or themes about replication work in empirical software engineering research (published between 1996 and 2012). ObjectiveIn this current article, our goal is to analyze and discuss the contents of the second set of studies about replications to increase our understanding of the current state of the work on replication in empirical software engineering research. MethodWe applied the systematic literature review method to build a systematic mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996-2012 complemented by manual and automatic search procedures that collected articles published in 2013. ResultsWe analyzed 37 papers reporting studies about replication published in the last 17years. These papers explore different topics related to concepts and classifications, presented guidelines, and discuss theoretical issues that are relevant for our understanding of replication in our field. We also investigated how these 37 papers have been cited in the 135 replication papers published between 1994 and 2012. ConclusionsReplication in SE still lacks a set of standardized concepts and terminology, which has a negative impact on the replication work in our field. To improve this situation, it is important that the SE research community engage on an effort to create and evaluate taxonomy, frameworks, guidelines, and methodologies to fully support the development of replications.

[1]  Per Runeson,et al.  A replicated experiment of usage-based and checklist-based reading , 2004, 10th International Symposium on Software Metrics, 2004. Proceedings..

[2]  Nazim H. Madhavji,et al.  Transitioning from lab studies to large-scale studies : Emerging results from a literal replication , 2010 .

[3]  Johan Per Fredrik Almqvist,et al.  Replication of Controlled Experiments in Empirical Software Engineering - A Survey , 2006 .

[4]  Natalia Juristo Juzgado,et al.  Replication of Software Engineering Experiments , 2010, LASER Summer School.

[5]  Natalia Juristo Juzgado,et al.  Using differences among replications of software engineering experiments to gain knowledge , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[6]  W. L. Kolb,et al.  A dictionary of the social sciences , 1964 .

[7]  Emilia Mendes,et al.  Investigating the effects of personality traits on pair programming in a higher education setting through a family of experiments , 2012, Empirical Software Engineering.

[8]  Marco Torchiano,et al.  On the effectiveness of screen mockups in requirements engineering: results from an internal replication , 2010, ESEM '10.

[9]  Per Runeson,et al.  Variation factors in the design and analysis of replicated controlled experiments , 2013, Empirical Software Engineering.

[10]  Terry Rout,et al.  Addressing the challenges of replications of surveys in software engineering research , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[11]  A. Brooks,et al.  Verification of results in software maintenance through external replication , 1994, Proceedings 1994 International Conference on Software Maintenance.

[12]  Silvia Mara Abrahão,et al.  A family of experiments to evaluate a functional size measurement procedure for Web applications , 2009, J. Syst. Softw..

[13]  Jeffrey C. Carver,et al.  Investigation of individual factors impacting the effectiveness of requirements inspections: a replicated experiment , 2014, Empirical Software Engineering.

[14]  Mario Piattini,et al.  A Controlled Experiment for Validating Class Diagram Structural Complexity Metrics , 2002, OOIS.

[15]  Edison Gonzalo Espinosa Gallardo.  Using configuration management and product line software paradigms to support the experimentation process in software engineering , 2012, 2012 Sixth International Conference on Research Challenges in Information Science (RCIS).

[16]  Filomena Ferrucci,et al.  Using Web Objects for Development Effort Estimation of Web Applications: A Replicated Study , 2011, PROFES.

[17]  Jeffrey C. Carver,et al.  The role of replications in Empirical Software Engineering , 2008, Empirical Software Engineering.

[18]  Rogério Eduardo Garcia,et al.  Packaging Controlled Experiments Using an Evolutionary Approach Based on Ontology(S) , 2011, SEKE.

[19]  Eliot R. Smith,et al.  Research methods in social relations , 1962 .

[20]  Lionel C. Briand,et al.  A Controlled Experiment for Evaluating Quality Guidelines on the Maintainability of Object-Oriented Designs , 2001, IEEE Trans. Software Eng..

[21]  Michelle Cartwright,et al.  A replication of the use of regression towards the mean (R2M) as an adjustment to effort estimation models , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[22]  C. Hendrick,et al.  Replications, strict replications, and conceptual replications: Are they important? , 1990 .

[23]  Martin Höst,et al.  Replication of an experiment on linguistic tool support for consolidation of requirements from multiple sources , 2011, Empirical Software Engineering.

[24]  Daniel M. Germán,et al.  Beyond Replication: An example of the potential benefits of replicability in the Mining of Software Repositories Community , 2010 .

[25]  Mario Piattini,et al.  Building UML class diagram maintainability prediction models based on early metrics , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[26]  G. Noblit,et al.  Meta-Ethnography: Synthesizing Qualitative Studies , 1988 .

[27]  Natalia Juristo Juzgado,et al.  A process for managing interaction between experimenters to get useful similar replications , 2013, Inf. Softw. Technol..

[28]  Omar S. Gómez,et al.  Replication , Reproduction and Re-analysis : Three ways for verifying experimental , 2010 .

[29]  Jeffrey C. Carver,et al.  A Framework for Software Engineering Experimental Replications , 2008, 13th IEEE International Conference on Engineering of Complex Computer Systems (iceccs 2008).

[30]  Fabio Q. B. da Silva,et al.  Replication of empirical studies in software engineering research: a systematic mapping study , 2012, Empirical Software Engineering.

[31]  A. Ehrenberg,et al.  The Design of Replicated Studies , 1993 .

[32]  M Wood,et al.  Replication of Experimental Results in Software Engineering , 2022 .

[33]  Jeffrey C. Carver.  Towards Reporting Guidelines for Experimental Replications: A Proposal , 2010 .

[34]  Gregorio Robles,et al.  Replicating MSR: A study of the potential replicability of papers published in the Mining Software Repositories proceedings , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[35]  Mario Piattini,et al.  A family of experiments to validate metrics for software process models , 2005, J. Syst. Softw..

[36]  Stephan Diehl,et al.  Comparison of similarity metrics for refactoring detection , 2011, MSR '11.

[37]  Miroslaw Staron,et al.  Empirical assessment of using stereotypes to improve comprehension of UML models: A set of experiments , 2006, J. Syst. Softw..

[38]  Jeffrey C. Carver,et al.  Replication types: towards a shared taxonomy , 2014, EASE '14.

[39]  Charles D. Knutson,et al.  A Case for Replication : Synthesizing Research Methodologies in Software Engineering , 2010 .

[40]  Tracy Hall,et al.  Program slicing-based cohesion measurement: the challenges of replicating studies using metrics , 2011, WETSoM '11.

[41]  Per Runeson,et al.  Experience from replicating empirical studies on prediction models , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[42]  Natalia Juristo Juzgado,et al.  Analysis of the influence of communication between researchers on experiment replication , 2006, ISESE '06.

[43]  Harald C. Gall,et al.  Replicating mining studies with SOFAS , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[44]  Sira Vegas,et al.  Design Patterns in Software Maintenance: An Experiment Replication at UPM - Experiences with the RESER'11 Joint Replication Project , 2011, 2011 Second International Workshop on Replication in Empirical Software Engineering Research.

[45]  Jeffrey C. Carver,et al.  Replicating software engineering experiments: addressing the tacit knowledge problem , 2002, Proceedings International Symposium on Empirical Software Engineering.

[46]  Giuseppe Scanniello,et al.  Does the Documentation of Design Pattern Instances Impact on Source Code Comprehension? Results from Two Controlled Experiments , 2011, 2011 18th Working Conference on Reverse Engineering.

[47]  Mario Piattini,et al.  Building measure-based prediction models for UML class diagram maintainability , 2007, Empirical Software Engineering.

[48]  Natalia Juristo Juzgado,et al.  Replications types in experimental disciplines , 2010, ESEM '10.

[49]  Victor R. Basili,et al.  Experimentation in software engineering , 1986, IEEE Transactions on Software Engineering.

[50]  Liming Zhu,et al.  Evaluating guidelines for empirical software engineering studies , 2006, ISESE '06.

[51]  Megan Squire.  A Replicable Infrastructure for Empirical Studies of Email Archives , 2013, 2013 3rd International Workshop on Replication in Empirical Software Engineering Research.

[52]  Thilo Mende,et al.  Replication of defect prediction studies: problems, pitfalls and recommendations , 2010, PROMISE '10.

[53]  Giuseppe Scanniello,et al.  A precise style for business process modelling: results from two controlled experiments , 2011, MODELS'11.

[54]  S. Schmidt.  Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences , 2009 .

[55]  Dietmar Pfahl,et al.  Reporting guidelines for controlled experiments in software engineering , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[56]  Michael A. La Sorte.  Replication as a Verification Technique in Survey Research: A Paradigm , 1972 .

[57]  Barbara A. Kitchenham,et al.  The role of replications in empirical software engineering—a word of warning , 2008, Empirical Software Engineering.

[58]  Fabio Q. B. da Silva,et al.  Team building criteria in software projects: A mix-method replicated study , 2013, Inf. Softw. Technol..

[59]  Ronnie E. S. Santos,et al.  Investigations about replication of empirical studies in software engineering: preliminary findings from a mapping study , 2014, EASE '14.

[60]  Natalia Juristo Juzgado,et al.  Report from the 2nd international workshop on replication in empirical software engineering research (RESER 2011) , 2012, SOEN.

[61]  Emilia Mendes,et al.  A replicated assessment of the use of adaptation rules to improve Web cost estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[62]  Mario Piattini,et al.  Evaluating the effect of composite states on the understandability of UML statechart diagrams , 2005, MoDELS'05.

[63]  Mario Piattini,et al.  Assessing the understandability of UML statechart diagrams with composite states—A family of empirical studies , 2009, Empirical Software Engineering.

[64]  Mario Piattini,et al.  The impact of structural complexity on the understandability of UML statechart diagrams , 2010, Inf. Sci..

[65]  Giuseppe Scanniello,et al.  Do professional developers benefit from design pattern documentation? a replication in the context of source code comprehension , 2012, MODELS'12.

[66]  N. Cartwright.  Replicability, Reproducibility, and Robustness: Comments on Harry Collins , 1991 .

[67]  James Miller,et al.  Replicating software engineering experiments: a poisoned chalice or the Holy Grail , 2005, Inf. Softw. Technol..

[68]  Natalia Juristo Juzgado,et al.  Understanding replication of experiments in software engineering: A classification , 2014, Inf. Softw. Technol..

[69]  Rahul Premraj,et al.  Network Versus Code Metrics to Predict Defects: A Replication Study , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[70]  Fabio Q. B. da Silva,et al.  The Effect of Reasoning Strategies on Success in Early Learning of Programming: Lessons Learned from an External Experiment Replication , 2010, EASE.

[71]  Gregory V. Wilson,et al.  On the difficulty of replicating human subjects studies in software engineering , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[72]  Jeffrey C. Carver,et al.  Knowledge-Sharing Issues in Experimental Software Engineering , 2004, Empirical Software Engineering.

[73]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[74]  Per Runeson,et al.  A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems , 2007, IEEE Transactions on Software Engineering.

[75]  Natalia Juristo Juzgado,et al.  Replications of software engineering experiments , 2013, Empirical Software Engineering.

[76]  Matthias M. Müller,et al.  Two controlled experiments concerning the comparison of pair programming to peer review , 2005, J. Syst. Softw..

[77]  Markus Borg,et al.  Confounding factors when conducting industrial replications in requirements engineering , 2013, 2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI).

[78]  Christian Bird,et al.  Report from the 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER 2013) , 2014, SOEN.

[79]  Carina Andersson,et al.  A replicated empirical study of a selection method for software reliability growth models , 2007, Empirical Software Engineering.

[80]  Mario Piattini,et al.  Evaluating performances of pair designing in industry , 2007, J. Syst. Softw..

[81]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[82]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[83]  Natalia Juristo Juzgado,et al.  The role of non-exact replications in software engineering experiments , 2011, Empirical Software Engineering.

[84]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[85]  Cleyton V. C. Magalhães,et al.  Towards a Taxonomy of Replications in Empirical Software Engineering Research: A Research Proposal , 2013 .

[86]  Filippo Lanubile,et al.  Investigating the use of tags in collaborative development environments: a replicated study , 2010, ESEM '10.

[87]  Jeffrey C. Carver,et al.  Replicated Studies: Building a Body of Knowledge about Software Reading Techniques , 2003, Lecture Notes on Empirical Software Engineering.

[88]  Mika V. Mäntylä,et al.  Rethinking Replication in Software Engineering: Can We See the Forest for the Trees? , 2010 .

[89]  Per Runeson,et al.  A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems , 2007, IEEE Transactions on Software Engineering.

[90]  Guilherme Horta Travassos,et al.  Evaluation of {model-based} testing techniques selection approaches: An external replication , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[91]  Maria Teresa Baldassarre,et al.  Assessing multiview framework (MF) comprehensibility and efficiency: A replicated experiment , 2006, Inf. Softw. Technol..

[92]  Günther Ruhe,et al.  A family of empirical studies to compare informal and optimization-based planning of software releases , 2006, ISESE '06.

[93]  M. Genero,et al.  Empirical validation of class diagram complexity metrics , 2001, SCCC 2001. 21st International Conference of the Chilean Computer Science Society.

[94]  Fabiano Cutigi Ferrari,et al.  Development of auxiliary functions: Should you be agile? An empirical assessment of pair programming and test-first programming , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[95]  Oscar Dieste,et al.  Hidden Evidence Behind Useless Replications , 2010 .

[96]  Mario Piattini,et al.  Assessing the influence of import-coupling on OCL expression maintainability: A cognitive theory-based perspective , 2010, Inf. Sci..

[97]  Natalia Juristo Juzgado,et al.  Report from the 1st international workshop on replication in empirical software engineering research (RESER 2010) , 2010, SOEN.