Systematic literature studies: Database searches vs. backward snowballing

Systematic studies of the literature can be done in different ways. In particular, different guidelines propose different first steps in their recommendations, e.g. start with search strings in different databases or start with the reference lists of a starting set of papers. In software engineering, the main recommended first step is using search strings in a number of databases, while in information systems, snowballing has been recommended as the first step. This paper compares the two different search approaches for conducting literature review studies. The comparison is conducted by searching for articles addressing “Agile practices in global software engineering”. The focus of the paper is on evaluating the two different search approaches. Despite the differences in the included papers, the conclusions and the patterns found in both studies are quite similar. The strengths and weaknesses of each first step are discussed separately and in comparison with each other. It is concluded that none of the first steps is outperforming the other, and the choice of guideline to follow, and hence the first step, may be context-specific, i.e. depending on the area of study.

[1]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[2]  Muhammad Ali Babar,et al.  Identifying relevant studies in software engineering , 2011, Inf. Softw. Technol..

[3]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[4]  Claes Wohlin,et al.  Agile Practices in Global Software Engineering - A Systematic Map , 2010, 2010 5th IEEE International Conference on Global Software Engineering.

[5]  Emilia Mendes,et al.  How Reliable Are Systematic Reviews in Empirical Software Engineering? , 2010, IEEE Transactions on Software Engineering.

[6]  Claes Wohlin,et al.  Global software engineering and agile practices: a systematic review , 2012, J. Softw. Evol. Process..

[7]  David Budgen,et al.  Evaluation and assessment in software engineering , 1997, J. Syst. Softw..

[8]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[9]  Claes Wohlin,et al.  Context in industrial software engineering research , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[10]  Pearl Brereton,et al.  Repeatability of systematic literature reviews , 2011, EASE.

[11]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[12]  Tony Gorschek,et al.  A method for evaluating rigor and industrial relevance of technology evaluations , 2011, Empirical Software Engineering.

[13]  T. Greenhalgh,et al.  Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources , 2005, BMJ : British Medical Journal.

[14]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[15]  LiGuo Huang,et al.  An empirical assessment of a systematic search process for systematic reviews , 2011, EASE.

[16]  Per Runeson,et al.  Reference-based search strategies in systematic reviews , 2009, EASE.

[17]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[18]  Muhammad Ali Babar,et al.  Systematic literature reviews in software engineering: Preliminary results from interviews with researchers , 2009, ESEM 2009.

[19]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..