Search Strategy to Update Systematic Literature Reviews in Software Engineering

[Context] Systematic Literature Reviews (SLRs) have been adopted within the Software Engineering (SE) domain for more than a decade to provide meaningful summaries of evidence on several topics. Many of these SLRs are now outdated, and there are no standard proposals on how to update SLRs in SE. [Objective] The goal of this paper is to provide recommendations on how to best to search for evidence when updating SLRs in SE. [Method] To achieve our goal, we compare and discuss outcomes from applying different search strategies to identifying primary studies in a previously published SLR update on effort estimation. [Results] The use of a single iteration forward snowballing with Google Scholar, and employing the original SLR and its primary studies as a seed set seems to be the most cost-effective way to search for new evidence when updating SLRs. [Conclusions] The recommendations can be used to support decisions on how to update SLRs in SE.

[1]  Claes Wohlin,et al.  When to Update Systematic Literature Reviews in Software Engineering , 2020, J. Syst. Softw..

[2]  Lionel C. Briand,et al.  An assessment and comparison of common software cost estimation modeling techniques , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[3]  Tim Menzies,et al.  When to use data from other projects for effort estimation , 2010, ASE.

[4]  Emilia Mendes,et al.  Using Forward Snowballing to update Systematic Reviews in Software Engineering , 2016, ESEM.

[5]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[6]  Emilia Mendes,et al.  Replicating studies on cross- vs single-company effort models using the ISBSG Database , 2008, Empirical Software Engineering.

[7]  Marta López,et al.  Formalizing a Systematic Review Updating Process , 2008, 2008 Sixth International Conference on Software Engineering Research, Management and Applications.

[8]  Bojan Cukic,et al.  Building a second opinion: learning cross-company data , 2013, PROMISE.

[9]  Ewan D. Tempero,et al.  A systematic review of software maintainability prediction and metrics , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[10]  Emilia Mendes,et al.  Effort estimation: how valuable is it for a web company to use a cross-company data set, compared to using its own single-company data set? , 2007, WWW '07.

[11]  Emilia Mendes,et al.  Cross- vs. within-company cost estimation studies revisited: an extended systematic review , 2014, EASE '14.

[12]  Claes Wohlin,et al.  Systematic literature reviews in software engineering , 2013, Inf. Softw. Technol..

[13]  Lionel C. Briand,et al.  A replicated assessment and comparison of common software cost modeling techniques , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[14]  Lina Maria Garcés Rodriguez,et al.  An Experience Report on Update of Systematic Literature Reviews , 2017, SEKE.

[15]  Tim Menzies,et al.  How to Find Relevant Data for Effort Estimation? , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[16]  Emilia Mendes,et al.  Web effort estimation: the value of cross-company data set compared to single-company data set , 2012, PROMISE '12.

[17]  Emilia Mendes,et al.  Further comparison of cross-company and within-company effort estimation models for Web applications , 2004 .

[18]  Emilia Mendes,et al.  Cross-company and single-company effort models using the ISBSG database: a further replicated study , 2006, ISESE '06.

[19]  Guilherme Horta Travassos,et al.  A systematic review of cross- vs. within- company cost estimation studies , 2006 .

[20]  Emilia Mendes,et al.  Using Chronological Splitting to Compare Cross- and Single-company Effort Models: Further Investigation , 2009, ACSC.

[21]  D. Ross Jeffery,et al.  A comparative study of two software development cost modeling techniques using multi-organizational and company-specific data , 2000, Inf. Softw. Technol..

[22]  Klaus Marius Hansen,et al.  Software ecosystems - A systematic literature review , 2013, J. Syst. Softw..

[23]  Isabella Wieczorek,et al.  How valuable is company-specific data compared to multi-company data for software cost estimation? , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[24]  D. Ross Jeffery,et al.  Using public domain metrics to estimate software development effort , 2001, Proceedings Seventh International Software Metrics Symposium.

[25]  Stephen G. MacDonell,et al.  A visual analysis approach to update systematic reviews , 2014, EASE '14.

[26]  Tina Klančar,et al.  Effort estimation in agile software development - A systematic literature review , 2017 .

[27]  Soumitra Dutta,et al.  Performance Evaluation of General and Company Specific Models in Software Development Effort Estimation , 1999 .

[28]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[29]  Martin J. Shepperd,et al.  Using Genetic Programming to Improve Software Effort Estimation Based on General Data Sets , 2003, GECCO.

[30]  Claes Wohlin,et al.  Agile Practices in Global Software Engineering - A Systematic Map , 2010, 2010 5th IEEE International Conference on Global Software Engineering.

[31]  Claes Wohlin,et al.  Second-generation systematic literature studies using snowballing , 2016, EASE.

[32]  Claes Wohlin,et al.  Global software engineering and agile practices: a systematic review , 2012, J. Softw. Evol. Process..

[33]  Konstantinos Manikas,et al.  Revisiting software ecosystems Research: A longitudinal literature study , 2016, J. Syst. Softw..

[34]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A tertiary study , 2010, Inf. Softw. Technol..

[35]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[36]  Claes Wohlin,et al.  On the Performance of Hybrid Search Strategies for Systematic Literature Reviews in Software Engineering , 2020, Inf. Softw. Technol..

[37]  Emilia Mendes,et al.  A replicated comparison of cross-company and within-company effort estimation models using the ISBSG database , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[38]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[39]  Emilia Mendes,et al.  Investigating the Use of Chronological Splitting to Compare Software Cross-company and Single-company Effort Predictions: A Replicated Study , 2009, EASE.

[40]  Onur Demirörs,et al.  Internal and External Software Benchmark Repository Utilization for Effort Estimation , 2011, 2011 Joint Conference of the 21st International Workshop on Software Measurement and the 6th International Conference on Software Process and Product Measurement.

[41]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[42]  Claes Wohlin,et al.  Investigating the Use of a Hybrid Search Strategy for Systematic Reviews , 2017, 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[43]  Jeffrey Earp,et al.  An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games , 2016, Comput. Educ..

[44]  Emilia Mendes,et al.  Investigating the use of chronological splitting to compare software cross-company and single-company effort predictions , 2008 .

[45]  Guilherme Horta Travassos,et al.  Cross versus Within-Company Cost Estimation Studies: A Systematic Review , 2007, IEEE Transactions on Software Engineering.

[46]  Thomas Grechenig,et al.  Systematic literature review on agile practices in global software development , 2017, Inf. Softw. Technol..

[47]  Thomas Zimmermann,et al.  Building Software Cost Estimation Models using Homogenous Data , 2007, ESEM 2007.

[48]  Barbara Kitchenham,et al.  A comparison of cross-company and within-company effort estimation models for Web applications , 2004, ICSE 2004.

[49]  Pearl Brereton,et al.  Reporting systematic reviews: Some lessons from a tertiary study , 2017, Inf. Softw. Technol..

[50]  Emilia Mendes,et al.  Cross-company vs. single-company web effort models using the Tukutuku database: An extended study , 2008, J. Syst. Softw..

[51]  Sérgio Soares,et al.  On the need to update systematic literature reviews , 2019, Inf. Softw. Technol..

[52]  André L. M. Santos,et al.  Six years of systematic literature reviews in software engineering: An updated tertiary study , 2011, Inf. Softw. Technol..

[53]  Mehwish Riaz Maintainability prediction of relational database-driven applications: A systematic review , 2012, EASE.

[54]  Tore Dybå,et al.  Evidence-based software engineering , 2016, Perspectives on Data Science for Software Engineering.

[55]  Xin Yao,et al.  Can cross-company data improve performance in software effort estimation? , 2012, PROMISE '12.

[56]  Mirko Perkusich,et al.  Effort Estimation in Agile Software Development: an Updated Review , 2018, SEKE.

[57]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.