Exploring Automatic Search in Digital Libraries: A Caution Guide for Systematic Reviewers

Search phase is considered as one of the most important steps in conducting secondary studies such as systematic literature reviews and mapping studies. In recent times, automatic search in digital libraries and academic search engines has been the preferred method of search phase execution for most software engineering related secondary studies. However, there are no previous studies that report or evaluate the secondary study relevant search features of these electronic data sources. We perform a feature analysis (screening mode) based evaluation of five widely used digital libraries (IEEE Xplore, ACM DL, SpringerLink, ScienceDirect and Wiley) in terms of their respective features required to support the search phase of secondary studies. We identify a total of 68 search related features and conduct a comprehensive exploration into their execution behaviors. The overall work presents a useful caution guide for systematic reviewers who plan to use the identified features for executing various search phase steps of their secondary studies.