Search. Review. Repeat? An empirical study of threats to replicating SLR searches

A systematic literature review (SLR) is an empirical method used to provide an overview of existing knowledge and to aggregate evidence within a domain. For computer science, several threats to the completeness of such reviews have been identified, leading to recommendations and guidelines on how to improve their quality. However, few studies address to what extent researchers can replicate an SLR. To conduct a replication, researchers have to first understand how the set of primary studies has been identified in the original study, and can ideally retrieve the same set when following the reported protocol. In this article, we focus on this initial step of a replication and report a two-fold empirical study: Initially, we performed a tertiary study using a sample of SLRs in computer science and identified what information that is needed to replicate the searches is reported. Based on the results, we conducted a descriptive, multi-case study on digital libraries to investigate to what extent these allow replications. The results reveal two threats to replications of SLRs: First, while researchers have improved the quality of their reports, relevant details are still missing—we refer to a reporting threat . Second, we found that some digital libraries are inconsistent in their query results—we refer to a searching threat . While researchers conducting a review can only overcome the first threat and the second may not be an issue for all kinds of replications, researchers should be aware of both threats when conducting, reviewing, and building on SLRs.