A critical appraisal of systematic reviews in software engineering from the perspective of the research questions asked in the reviews

After a seminal article introducing-evidence based software engineering in 2004, systematic reviews (SR) have been increasingly used as a method for conducting secondary studies in software engineering. Our goal is to critically appraise the use of SR in software engineering with respect to the research questions asked and the ways the questions were used in the reviews. We analyzed 53 literature reviews that had been collected in two published tertiary studies. We found that over 65% of the research questions asked in the reviews were exploratory and only 15% investigated causality questions. We concluded that there is a need for a consistent use of terminology to classify secondary studies and that reports of literature reviews should follow reporting guidelines to support assessment and comparison.