Benefitting from the Grey Literature in Software Engineering Research

Researchers generally place the most trust in peer-reviewed, published information, such as journals and conference papers. By contrast, software engineering (SE) practitioners typically do not have the time, access or expertise to review and benefit from such publications. As a result, practitioners are more likely to turn to other sources of information that they trust, e.g., trade magazines, online blog-posts, survey results or technical reports, collectively referred to as Grey Literature (GL). Furthermore, practitioners also share their ideas and experiences as GL, which can serve as a valuable data source for research. While GL itself is not a new topic in SE, using, benefitting and synthesizing knowledge from the GL in SE is a contemporary topic in empirical SE research and we are seeing that researchers are increasingly benefitting from the knowledge available within GL. The goal of this chapter is to provide an overview to GL in SE, together with insights on how SE researchers can effectively use and benefit from the knowledge and evidence available in the vast amount of GL.

[1]  Vahid Garousi,et al.  Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review , 2016, Inf. Softw. Technol..

[2]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[3]  Kai Petersen,et al.  Survey Guidelines in Software Engineering: An Annotated Review , 2016, ESEM.

[4]  Julia A. Kelly,et al.  The types, frequencies, and findability of disciplinary grey literature within prominent subject databases and academic institutional repositories , 2018 .

[5]  Dominique Arrouays,et al.  Is ‘grey literature’ a reliable source of data to characterize soils at the scale of a region? A case study in a maritime pine forest in southwestern France , 2010 .

[6]  Austen Rainer,et al.  Using argumentation theory to analyse software practitioners' defeasible evidence, inference and belief , 2017, Inf. Softw. Technol..

[7]  Ricardo Colomo-Palacios,et al.  Serverless computing: a multivocal literature review , 2018 .

[8]  Ralf Krestel,et al.  Topic Shifts in StackOverflow: Ask it Like Socrates , 2016, NLDB.

[9]  Premkumar T. Devanbu,et al.  Belief & Evidence in Empirical Software Engineering , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[10]  Mika Mäntylä,et al.  Gamification of Software Testing - An MLR , 2016, PROFES.

[11]  Sung-Hyon Myaeng,et al.  Detecting Experiences from Weblogs , 2010, ACL.

[12]  Ricardo Terra,et al.  Monorepos: A Multivocal Literature Review , 2018, ArXiv.

[13]  Vahid Garousi,et al.  Smells in software test code: A survey of knowledge in industry and academia , 2018, J. Syst. Softw..

[14]  A. Aberkane Exploring Ethics in Requirements Engineering , 2018 .

[15]  Vahid Garousi,et al.  Industry-academia collaborations in software engineering: An empirical analysis of challenges, patterns and anti-patterns in research projects , 2017, EASE.

[16]  Bertrum H. MacDonald,et al.  Assessing the Diffusion and Impact of Grey Literature Published by International Intergovernmental Scientific Groups: The Case of the Gulf of Maine Council on the Marine Environment , 2007 .

[17]  Pasi Kuvaja,et al.  Relationship of DevOps to Agile, Lean and Continuous Deployment - A Multivocal Literature Review Study , 2016, PROFES.

[18]  Sarah Bonato Searching the grey literature: a handbook for searching reports, working papers, and other unpublished research. , 2018 .

[19]  Larry A. Thompson Grey Literature in Engineering , 2001 .

[20]  Vahid Garousi,et al.  Guidelines for including the grey literature and conducting multivocal literature reviews in software engineering , 2017, Inf. Softw. Technol..

[21]  Ricardo Colomo Palacios,et al.  A Multivocal Literature Review on the use of DevOps for e-Learning systems , 2018, TEEM.

[22]  Austen Rainer,et al.  Persuading developers to "buy into" software process improvement: a local opinion and empirical evidence , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[23]  Pearl Brereton,et al.  Evidence-Based Software Engineering and Systematic Reviews , 2015 .

[24]  Daniela Cruzes,et al.  Analyzing the Impact of Beliefs in Software Project Practices , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[25]  Helen Sharp,et al.  Using ethnographic methods in software engineering research , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[26]  Vahid Garousi,et al.  Guidelines for conducting multivocal literature reviews in software engineering , 2017 .

[27]  Josef Spillner,et al.  A mixed-method empirical study of Function-as-a-Service software development in industrial practice , 2018, PeerJ Prepr..

[28]  Josef Spillner,et al.  A mixed-method empirical study of Function-as-a-Service software development in industrial practice , 2018, PeerJ Prepr..

[29]  M. Patton Qualitative research and evaluation methods , 1980 .

[30]  Vahid Garousi,et al.  The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature , 2016, EASE.

[31]  Tom DeMarco,et al.  Peopleware: Productive Projects and Teams , 1987 .

[32]  Ko Fujimura,et al.  Discovering Association Rules on Experiences from Large-Scale Blog Entries , 2009, ECIR.

[33]  Martin Oberhofer,et al.  Knowledge Discovery in the Blogosphere: Approaches and Challenges , 2010, IEEE Internet Computing.

[34]  Diomidis Spinellis,et al.  Word Embeddings for the Software Engineering Domain , 2018, 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR).

[35]  R. Adams,et al.  Shades of Grey: Guidelines for Working with the Grey Literature in Systematic Reviews for Management and Organizational Studies , 2017 .

[36]  Austen Rainer,et al.  Investigating developers' email discussions during decision-making in Python language evolution , 2017, EASE.

[37]  Frank Wm. Tompa,et al.  Seeking Stable Clusters in the Blogosphere , 2007, VLDB.

[38]  Vahid Garousi,et al.  Choosing the Right Test Automation Tool: a Grey Literature Review of Practitioner Sources , 2017, EASE.

[39]  Austen Rainer,et al.  Toward the use of blog articles as a source of evidence for software engineering research , 2017, EASE.

[40]  Vahid Garousi,et al.  Software test maturity assessment and test process improvement: A multivocal literature review , 2017, Inf. Softw. Technol..

[41]  Tom DeMarco,et al.  Peopleware: Productive Projects and Teams (3rd Edition) , 2013 .

[42]  Emad Shihab,et al.  What are mobile developers asking about? A large scale study using stack overflow , 2016, Empirical Software Engineering.

[43]  Ivans Kulesovs iOS Applications Testing , 2015 .

[44]  Willem-Jan van den Heuvel,et al.  The pains and gains of microservices: A Systematic grey literature review , 2018, J. Syst. Softw..

[45]  Christoph Treude,et al.  Measuring API documentation on the web , 2011, Web2SE '11.

[46]  Gregory Tassey,et al.  Prepared for what , 2007 .

[47]  Christoph Treude,et al.  Blogging developer knowledge: Motivations, challenges, and future directions , 2013, 2013 21st International Conference on Program Comprehension (ICPC).

[48]  Leif Singer,et al.  The (R) Evolution of social media in software engineering , 2014, FOSE.

[49]  Lionel Briand,et al.  Embracing the Engineering Side of Software Engineering , 2012, IEEE Softw..

[50]  Rory V. O'Connor,et al.  A multivocal literature review on serious games for software process standards education , 2018, Comput. Stand. Interfaces.

[51]  Michele Tarsilla Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[52]  Joanna Szurmak,et al.  Beyond Grey Literature: How Grey Questions Can Drive Research , 2002 .

[53]  C. Lee Giles,et al.  Accessibility of information on the Web , 2000, INTL.

[54]  Pekka Abrahamsson,et al.  “Failures” to be celebrated: an analysis of major pivots of software startups , 2016, Empirical Software Engineering.

[55]  Richard T. Vidgen,et al.  An exploration of technical debt , 2013, J. Syst. Softw..

[56]  Marcus A. Banks Towards a continuum of scholarship: The eventual collapse of the distinction between grey and non-grey literature , 2006 .

[57]  Guilherme Horta Travassos,et al.  Characterizing DevOps by Hearing Multiple Voices , 2016, SBES '16.

[58]  Cleo Pappas,et al.  Grey Literature: Its Emerging Importance , 2011 .

[59]  Daniela Luzi,et al.  Trends and evolution in the development of grey literature: a review , 2000 .

[60]  Nicole Novielli,et al.  How to ask for technical help? Evidence-based guidelines for writing questions on Stack Overflow , 2017, Inf. Softw. Technol..

[61]  Ricardo Colomo Palacios,et al.  DevSecOps: A Multivocal Literature Review , 2017, SPICE.

[62]  Pier Luca Lanzi,et al.  Mining interesting knowledge from weblogs: a survey , 2005, Data Knowl. Eng..

[63]  Austen Rainer,et al.  Using blog‐like documents to investigate software practice: Benefits, challenges, and research directions , 2019, J. Softw. Evol. Process..

[64]  Kentaro Inui,et al.  Experience Mining: Building a Large-Scale Database of Personal Experiences and Opinions from Web Documents , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[65]  Anna H. Perrault,et al.  The Impact of Grey Literature in Advancing Global Karst Research: An Information Needs Assessment for a Globally Distributed Interdisciplinary Community , 2007 .

[66]  Frederick P. Brooks,et al.  The Mythical Man-Month: Essays on Softw , 1978 .

[67]  Casper Lassenius,et al.  Benefits and Challenges of Adopting the Scaled Agile Framework (SAFe): Preliminary Results from a Multivocal Literature Review , 2018, PROFES.

[68]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[69]  Daniela Cruzes,et al.  Recommended Steps for Thematic Synthesis in Software Engineering , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[70]  Katsumi Tanaka,et al.  Mining and Visualizing Local Experiences from Blog Entries , 2006, DEXA.

[71]  Vahid Garousi,et al.  Practical relevance of software engineering research: synthesizing the community’s voice , 2018, Empirical Software Engineering.

[72]  Dominic J. Farace Rise of the phoenix: A review of new forms and exploitations of grey literature , 1997 .

[73]  Margaret-Anne D. Storey,et al.  Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube , 2015, 2015 IEEE 23rd International Conference on Program Comprehension.

[74]  Vahid Garousi,et al.  When and what to automate in software testing? A multi-vocal literature review , 2016, Inf. Softw. Technol..

[75]  Elahe Rahimtoroghi,et al.  Identifying Narrative Clause Types in Personal Stories , 2014, SIGDIAL Conference.