Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies

Despite recognition that database search alone is inadequate even within the health sciences, it appears that reviewers in fields that have adopted systematic review are choosing to rely primarily, or only, on database search for information retrieval. This commentary reminds readers of factors that call into question the appropriateness of default reliance on database searches particularly as systematic review is adapted for use in new and lower consensus fields. It then discusses alternative methods for information retrieval that require development, formalisation, and evaluation. Our goals are to encourage reviewers to reflect critically and transparently on their choice of information retrieval methods and to encourage investment in research on alternatives.

[1]  H. Tobi,et al.  Which standards from which disciplines? A test of systematic review for designing interdisciplinary evaluations , 2017 .

[2]  Taina Eriksson,et al.  Methodological issues in dynamic capabilities research – a critical review , 2013 .

[3]  R. Plummer,et al.  Adaptive Comanagement and Its Relationship to Environmental Governance , 2013 .

[4]  A. Booth.  The number needed to retrieve: a practically useful measure of information retrieval? , 2006, Health information and libraries journal.

[5]  L. Preston,et al.  Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking , 2015, Systematic Reviews.

[6]  I. McKeith,et al.  Diagnosing Mild Cognitive Impairment (MCI) in clinical trials: a systematic review , 2013, BMJ Open.

[7]  Sebastian K. Boell,et al.  On being ‘systematic’ in literature reviews in IS , 2015, J. Inf. Technol..

[8]  M Sandelowski,et al.  Focus on qualitative methods. Qualitative metasynthesis: issues and techniques. , 1997, Research in nursing & health.

[9]  Peter Tugwell,et al.  How to do a good systematic review of effects in international development: a tool kit , 2012 .

[10]  Robin Holt,et al.  Knowledge, learning and small firm growth: A systematic review of the evidence , 2007 .

[11]  M. Kirkevold,et al.  Instruments Measuring Nursing Staff Competence in Community Health Care , 2013 .

[12]  C. Chew‐Graham,et al.  PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews , 2014, BMC Health Services Research.

[13]  M. Dixon-Woods,et al.  Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups , 2006 .

[14]  P. Kearney,et al.  The effect of moderate gestational alcohol consumption during pregnancy on speech and language outcomes in children: a systematic review , 2014, Systematic Reviews.

[15]  Claire Stansfield,et al.  Search wide, dig deep: literature searching for qualitative research. An analysis of the publication formats and information sources used for four systematic reviews in public health , 2014, Research synthesis methods.

[16]  James D. Ford,et al.  Systematic review approaches for climate change adaptation research , 2015, Regional Environmental Change.

[17]  Andrew Booth,et al.  Desperately Seeking Dissonance , 2013, Qualitative health research.

[18]  J. Magarey.  Elements of a systematic review. , 2001, International journal of nursing practice.

[19]  T. Crane,et al.  A systematic review of local vulnerability to climate change in developing country agriculture , 2017 .

[20]  Susan R. Wilson,et al.  Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews , 1998, BMJ.

[21]  Ginny Brunton,et al.  Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example , 2014, Systematic Reviews.

[22]  Gerald Gartlehner,et al.  Semi-automating the manual literature search for systematic reviews increases efficiency. , 2010, Health information and libraries journal.

[23]  Andrew Booth,et al.  Towards a methodology for cluster searching to provide conceptual and contextual “richness” for systematic reviews of complex interventions: case study (CLUSTER) , 2013, BMC Medical Research Methodology.

[24]  Jacqueline Davis,et al.  A systematic method for search term selection in systematic reviews , 2014, Research synthesis methods.

[25]  Natasha Kuruppu,et al.  Barriers to reducing climate enhanced disaster risks in Least Developed Country-Small Islands through anticipatory adaptation , 2015 .

[26]  David Ogilvie,et al.  Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews , 2014, Research synthesis methods.

[27]  James Hartley,et al.  An Evaluation of Structured Abstracts in Journals Published by the British Psychological Society , 1998 .

[28]  Tim Harries,et al.  The Anticipated Emotional Consequences of Adaptive Behaviour—Impacts on the Take-up of Household Flood-Protection Measures , 2012 .

[29]  H. Stelfox,et al.  Capture-mark-recapture as a tool for estimating the number of articles available for systematic reviews in critical care medicine. , 2013, Journal of critical care.

[30]  Klaus Glenk,et al.  Insurance, prevention or just wait and see? Public preferences for water management strategies in the context of climate change. , 2010 .

[31]  Kate Ashton,et al.  Effectiveness of Search Strategies for Qualitative Research About Barriers and Facilitators of Program Delivery , 2011, Evaluation & the health professions.

[32]  T. Greenhalgh,et al.  Realist review - a new method of systematic review designed for complex policy interventions , 2005, Journal of health services research & policy.

[33]  David Ogilvie,et al.  Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? , 2005, Journal of Epidemiology and Community Health.

[34]  David Evans,et al.  Database searches for qualitative research. , 2002, Journal of the Medical Library Association : JMLA.

[35]  J. Lexchin.  Sponsorship bias in clinical research. , 2012, The International journal of risk & safety in medicine.

[36]  Gerta Rücker,et al.  Boosting qualifies capture-recapture methods for estimating the comprehensiveness of literature searches for systematic reviews. , 2011, Journal of clinical epidemiology.

[37]  Gillian Frances Menzies,et al.  Designing domestic buildings for future summers: Attitudes and opinions of building professionals , 2012 .

[38]  Lindsey Jones,et al.  Strengthening the food systems governance evidence base: Supporting commensurability of research through a systematic review of methods , 2016 .

[39]  E. Lambin,et al.  Responding to Globalization: Impacts of Certification on Colombian Small-Scale Coffee Growers , 2013 .

[40]  Helen R. Bayliss,et al.  Shades of grey: Two forms of grey literature important for reviews in conservation , 2015 .

[41]  T. de Wet,et al.  Purity or pragmatism? Reflecting on the use of systematic review methodology in development , 2012 .

[42]  Tina Miller,et al.  Finding qualitative research: an evaluation of search strategies , 2004, BMC medical research methodology.

[43]  Helen R. Bayliss,et al.  The reliability of evidence review methodology in environmental science and conservation , 2016 .

[44]  Andrew Booth,et al.  Beyond PICO , 2012, Qualitative health research.

[45]  N. Gunawardena,et al.  A review of the Sri Lankan health-sector response to intimate partner violence: looking back, moving forward , 2015, WHO South-East Asia journal of public health.

[46]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[47]  P. Bower,et al.  The European General Practice Research Network presents a comprehensive definition of multimorbidity in family medicine and long term care, following a systematic review of relevant literature. , 2013, Journal of the American Medical Directors Association.

[48]  Deborah Finfgeld-Connett,et al.  Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. , 2013, Journal of advanced nursing.

[49]  Martin Offringa,et al.  Definitions and measurement of chronic health conditions in childhood: a systematic review. , 2007, JAMA.

[50]  A. Harden,et al.  Methods for the thematic synthesis of qualitative research in systematic reviews , 2008, BMC medical research methodology.

[51]  A. Cardona,et al.  An e-mail survey identified unpublished studies for systematic reviews. , 2006, Journal of Clinical Epidemiology.

[52]  Heitor Augustus Xavier Costa,et al.  Systematic literature review supported by information retrieval techniques: A case study , 2014, 2014 XL Latin American Computing Conference (CLEI).

[53]  James J. Porter,et al.  What do we know about UK household adaptation to climate change? A systematic review , 2014, Climatic Change.

[54]  Allan Hanbury,et al.  DASyR(IR) - document analysis system for systematic reviews (in Information Retrieval) , 2015, 2015 13th International Conference on Document Analysis and Recognition (ICDAR).

[55]  J. Kavanagh,et al.  Mixed Methods Synthesis: A Worked Example , 2012 .

[56]  A. Booth.  Searching for qualitative research for inclusion in systematic reviews: a structured methodological review , 2016, Systematic Reviews.

[57]  Lee Hooper,et al.  Why Are Medical and Health-Related Studies Not Being Published? A Systematic Review of Reasons Given by Investigators , 2014, PloS one.

[58]  A. Kazanjian,et al.  BEYOND MEDLINE , 2003, International Journal of Technology Assessment in Health Care.

[59]  Maren Duvendack,et al.  The benefits and challenges of using systematic reviews in international development research , 2012 .

[60]  Andrew Booth,et al.  How much searching is enough? Comprehensive versus optimal retrieval for technology assessments , 2010, International Journal of Technology Assessment in Health Care.