The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature

Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey literature in addition to the published (formal) literature. Only a few MLRs have been published in SE so far. We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs? To answer these RQs, we sample a few example SLRs and MLRs and identify the missing and the gained knowledge due to excluding or including the grey literature. We find that (1) grey literature can give substantial benefits in certain areas of SE, and that (2) the inclusion of grey literature brings forward certain challenges as evidence in them is often experience and opinion based. Given these conflicting viewpoints, the authors are planning to prepare systematic guidelines for performing MLRs in SE.

[1]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[2]  AvgeriouParis,et al.  The financial aspect of managing technical debt , 2015 .

[3]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[4]  Girish Suryanarayana,et al.  Refactoring for Software Design Smells: Managing Technical Debt , 2014 .

[5]  Richard T. Vidgen,et al.  An exploration of technical debt , 2013, J. Syst. Softw..

[6]  Peng Liang,et al.  A systematic mapping study on technical debt and its management , 2015, J. Syst. Softw..

[7]  M Egger,et al.  Grey literature in meta-analyses of randomized trials of health care interventions. , 2007, The Cochrane database of systematic reviews.

[8]  W. Whyte,et al.  Participatory Action Research , 1989 .

[9]  Chris Sterling,et al.  Managing Software Debt: Building for Inevitable Change , 2010 .

[10]  Sally Hopewell,et al.  Grey Literature and Systematic Reviews , 2006 .

[11]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[12]  Rory V. O'Connor,et al.  The situational factors that affect the software development process: Towards a comprehensive reference framework , 2012, Inf. Softw. Technol..

[13]  Kent Beck,et al.  Extreme Programming Explained: Embrace Change (2nd Edition) , 2004 .

[14]  Ivans Kulesovs iOS Applications Testing , 2015 .

[15]  Mika Mäntylä,et al.  Using metrics in Agile and Lean Software Development - A systematic literature review of industrial studies , 2015, Inf. Softw. Technol..

[16]  Jeff Sutherland,et al.  The Scrum Guide , 2012 .

[17]  Richard F. Elmore,et al.  Comment on “Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method” , 1991 .

[18]  Dwayne Van Eerd,et al.  Searching for grey literature for systematic reviews: challenges and benefits , 2014, Research synthesis methods.

[19]  Apostolos Ampatzoglou,et al.  The financial aspect of managing technical debt: A systematic literature review , 2015, Inf. Softw. Technol..

[20]  Isaac Pentinmaki,et al.  Review of "Lean software development: an agile tookit" by Mary and Tom Poppendieck. Addison Wesley 2003. , 2004, SOEN.

[21]  M. Borenstein,et al.  Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , 2006 .

[22]  Mary Poppendieck,et al.  Lean Software Development: An Agile Toolkit , 2003 .

[23]  Michael Quinn Patton,et al.  Towards Utility in Reviews of Multivocal Literatures , 1991 .

[24]  Richard T. Vidgen,et al.  A Consolidated Understanding of Technical debt , 2012, ECIS.

[25]  Claes Wohlin,et al.  Context in industrial software engineering research , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[26]  Vahid Garousi,et al.  Graphical user interface (GUI) testing: Systematic mapping and repository , 2013, Inf. Softw. Technol..

[27]  Rodney T. Ogawa,et al.  Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method , 1991 .

[28]  Affan Yasin,et al.  On the Quality of Grey Literature and its use in Information Synthesis during Systematic Literature Reviews , 2012 .

[29]  K. A. Hayden,et al.  State-of-the-evidence reviews: advantages and challenges of including grey literature. , 2006, Worldviews on evidence-based nursing.

[30]  Dave Nicolette Software Development Metrics , 2015 .

[31]  Navneesh Garg Test Automation using Selenium WebDriver with Java: Step by Step Guide , 2014 .

[32]  Emilia Mendes,et al.  A Systematic Literature Review of Software Process Improvement in Small and Medium Web Companies , 2009, FGIT-ASEA.

[33]  Girish Suryanarayana,et al.  Chapter 1 – Technical Debt , 2015 .

[34]  Claes Wohlin,et al.  Software Engineering Research under the Lamppost , 2013, ICSOFT.

[35]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .