Systematic literature reviews in software engineering - A systematic literature review

Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference. Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence. Method: We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings. Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4. Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners.

[1]  Per Runeson,et al.  What do we know about defect detection methods? [software testing] , 2006, IEEE Software.

[2]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[3]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[4]  Marco Torchiano,et al.  Overlooked aspects of COTS-based development , 2004, IEEE Software.

[5]  Emilia Mendes,et al.  A systematic review of Web engineering research , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[6]  Anette C. Lien,et al.  A survey on software estimation in the Norwegian industry , 2004 .

[7]  Natalia Juristo Juzgado,et al.  Reviewing 25 Years of Testing Technique Experiments , 2004, Empirical Software Engineering.

[8]  Venkataraman Ramesh,et al.  Research in software engineering: an analysis of the literature , 2002, Inf. Softw. Technol..

[9]  Marvin V. Zelkowitz,et al.  Experimental Models for Validating Technology , 1998, Computer.

[10]  Natalia Juristo Juzgado,et al.  In Search of What We Experimentally Know about Unit Testing , 2006, IEEE Software.

[11]  Guilherme Horta Travassos,et al.  Evaluation Approaches for Software Architectural Documents: a Systematic Review , 2006, CIbSE.

[12]  Magne Jørgensen,et al.  Software effort estimation terminology: The tower of Babel , 2006, Inf. Softw. Technol..

[13]  Tore Dybå,et al.  A Systematic Review of Theory Use in Software Engineering Experiments , 2007, IEEE Transactions on Software Engineering.

[14]  Pearl Brereton,et al.  Lessons learnt Undertaking a Large-scale Systematic Literature Review , 2008, EASE.

[15]  Sajjad Mahmood,et al.  A survey of component based system quality assurance and assessment , 2005, Inf. Softw. Technol..

[16]  Carolyn Mair,et al.  The consistency of empirical comparisons of regression and analogy-based software project cost prediction , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[17]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[18]  Magne Jørgensen,et al.  Forecasting of software development work effort: Evidence on expert judgement and formal models , 2007 .

[19]  Jeffrey V. Nickerson,et al.  Hands-on, simulated, and remote laboratories: A comparative literature review , 2006, CSUR.

[20]  Sajjad Mahmood,et al.  Survey of component-based software development , 2007, IET Softw..

[21]  Marvin V. Zelkowitz,et al.  EXPERIMENTAL MODELS FOR VALIDATING COMPUTER TECHNOLOGY , 2001 .

[22]  Marvin V. Zelkowitz,et al.  Experimental validation in software engineering , 1997, Inf. Softw. Technol..

[23]  L. Green,et al.  Hands on! , 2008, MLO: medical laboratory observer.

[24]  Venkataraman Ramesh,et al.  An analysis of research in computing disciplines , 2004, CACM.

[25]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..

[26]  Tore Dybå,et al.  A systematic review of statistical power in software engineering experiments , 2006, Inf. Softw. Technol..

[27]  Grigori Melnik,et al.  On the success of empirical studies in the international conference on software engineering , 2006, ICSE.

[28]  Daniel Galin,et al.  Are CMM Program Investments Beneficial? Analyzing Past Studies , 2006, IEEE Software.

[29]  Barbara A. Kitchenham,et al.  Combining empirical results in software engineering , 1998, Inf. Softw. Technol..

[30]  James Miller,et al.  Applying meta-analytical procedures to software engineering experiments , 2000, J. Syst. Softw..

[31]  Kjetil Moløkken-Østvold,et al.  A survey on software estimation in the Norwegian industry , 2004, 10th International Symposium on Software Metrics, 2004. Proceedings..

[32]  Venkataraman Ramesh,et al.  Research in computer science: an empirical study , 2004, J. Syst. Softw..

[33]  D. Aaker,et al.  The sophistication of naive modeling , 1987 .

[34]  Daniel Galin,et al.  Do SQA programs work - CMM works. a meta analysis , 2005, IEEE International Conference on Software - Science, Technology & Engineering (SwSTE'05).

[35]  Tore Dybå,et al.  Teaching evidence-based software engineering to university students , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[36]  Guilherme Horta Travassos,et al.  Cross versus Within-Company Cost Estimation Studies: A Systematic Review , 2007, IEEE Transactions on Software Engineering.

[37]  Guilherme Horta Travassos,et al.  A systematic review of cross- vs. within- company cost estimation studies , 2006 .

[38]  Claes Wohlin,et al.  Capture-recapture in software inspections after 10 years research--theory, evaluation and application , 2004, J. Syst. Softw..

[39]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.

[40]  James Miller.  Can results from software engineering experiments be safely combined? , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[41]  Paul Lukowicz,et al.  Experimental evaluation in computer science: A quantitative study , 1995, J. Syst. Softw..

[42]  Magne Jørgensen,et al.  A review of studies on expert estimation of software development effort , 2004, J. Syst. Softw..

[43]  Will Hayes,et al.  Research synthesis in software engineering: a case for meta-analysis , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).