Appraising Qualitative Research for Evidence Syntheses: A Compendium of Quality Appraisal Tools

As the movement toward evidence-based health policy continues to emphasize the importance of including patient and public perspectives, syntheses of qualitative health research are becoming more common. In response to the focus on independent assessments of rigor in these knowledge products, over 100 appraisal tools for assessing the quality of qualitative research have been developed. The variety of appraisal tools exhibit diverse methods and purposes, reflecting the lack of consensus as to what constitutes appropriate quality criteria for qualitative research. It is a daunting task for those without deep familiarity of the field to choose the best appraisal tool for their purpose. This article provides a description of the structure, content, and objectives of existing appraisal tools for those wanting to evaluate primary qualitative research for a qualitative evidence synthesis. We then discuss common features of appraisal tools and examine their implications for evidence synthesis.

[1]  J. A. Smith,et al.  The problem of appraising qualitative research , 2004, Quality and Safety in Health Care.

[2]  David R. Jones,et al.  Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods , 2007, Journal of health services research & policy.

[3]  Lucy Yardley,et al.  Demonstrating validity in qualitative psychology , 2007 .

[4]  M. Vanstone,et al.  Prostate Cancer Patient Perspectives on the Use of Information in Treatment Decision-Making: A Systematic Review and Qualitative Meta-synthesis. , 2017, Ontario health technology assessment series.

[5]  Mary Godfrey,et al.  An evaluation tool to assess the quality of qualitative research studies , 2004 .

[6]  Margarete Sandelowski,et al.  Creating Metasummaries of Qualitative Findings , 2003, Nursing research.

[7]  Stephen J. Gentles,et al.  Critical Approach to Reflexivity in Grounded Theory , 2014 .

[8]  Louise A. Jensen,et al.  Meta-Synthesis of Qualitative Findings , 1996 .

[9]  Christopher Carroll,et al.  Qualitative evidence synthesis to improve implementation of clinical guidelines , 2017, British Medical Journal.

[10]  Souraya Sidani,et al.  Handbook for Synthesizing Qualitative Research , 2008 .

[11]  Ruth Garside,et al.  Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how? , 2014 .

[12]  A. Booth,et al.  Should We Exclude Inadequately Reported Studies From Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews , 2012, Qualitative health research.

[13]  Norman K. Denzin,et al.  The elephant in the living room: or extending the conversation about the politics of evidence , 2009 .

[14]  M. Sandelowski.  Focus on Research Methods Whatever Happened to Qualitative Description? , 2022 .

[15]  Andrew Booth,et al.  COREQ (Consolidated Criteria for Reporting Qualitative Studies) , 2014 .

[16]  K. Melia.  Recognizing Quality in Qualitative Research , 2010 .

[17]  Zachary Munn,et al.  Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation , 2015, International journal of evidence-based healthcare.

[18]  Andrew Booth,et al.  Using Qualitative Evidence in Decision Making for Health and Social Interventions: An Approach to Assess Confidence in Findings from Qualitative Evidence Syntheses (GRADE-CERQual) , 2015, PLoS medicine.

[19]  Denis Walsh,et al.  Appraising the quality of qualitative research. , 2006, Midwifery.

[20]  Andrew Booth,et al.  Quality assessment of qualitative evidence for systematic review and synthesis: Is it meaningful, and if so, how should it be performed? , 2015, Research synthesis methods.

[21]  Bridget C. O’Brien,et al.  Standards for Reporting Qualitative Research: A Synthesis of Recommendations , 2014, Academic medicine : journal of the Association of American Medical Colleges.

[22]  A. Harden,et al.  Methods for the thematic synthesis of qualitative research in systematic reviews , 2008, BMC medical research methodology.

[23]  Rosaline S Barbour,et al.  Evaluating and synthesizing qualitative research: the need to develop a distinctive approach. , 2003, Journal of evaluation in clinical practice.

[24]  Brynjulf Stige,et al.  Toward an Agenda for Evaluation of Qualitative Research , 2009, Qualitative health research.

[25]  D. Cook,et al.  Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? Evidence-Based Medicine Working Group. , 2000, JAMA.

[26]  Karin Hannes,et al.  A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers , 2012 .

[27]  S. Thorne.  Metasynthetic Madness , 2016, Qualitative health research.

[28]  Jennie Popay,et al.  Directly observed therapy and tuberculosis: how can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. , 2007, Journal of advanced nursing.

[29]  L. Belgrave,et al.  How do we Talk to Each other? Writing Qualitative Research for Quantitative Readers , 2002, Qualitative health research.

[30]  Janice M. Morse,et al.  Aggregating Qualitative Findings: An Approach to Theory Development , 1994 .

[31]  M. Giacomini,et al.  Type 2 diabetes patients’ and providers’ differing perspectives on medication nonadherence: a qualitative meta-synthesis , 2015, BMC Health Services Research.

[32]  Iveta Simera,et al.  A proposed framework for developing quality assessment tools , 2017, Systematic Reviews.

[33]  T. Koch,et al.  Reconceptualizing rigour: the case for reflexivity. , 1998, Journal of advanced nursing.

[34]  James Thomas,et al.  Bmc Medical Research Methodology Methods for the Synthesis of Qualitative Research: a Critical Review , 2022 .

[35]  M. Dixon-Woods,et al.  The problem of appraising qualitative research. , 2004 .

[36]  S. Downe.  Metasynthesis: a guide to knitting smoke , 2008 .

[37]  Rosaline S Barbour,et al.  Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? , 2001, BMJ : British Medical Journal.

[38]  J. Newton,et al.  An appraisal of the quality of published qualitative dental research. , 2011, Community dentistry and oral epidemiology.

[39]  Alex J. Sutton,et al.  Synthesizing qualitative research: a review of published reports , 2007 .

[40]  P. Sainsbury,et al.  Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. , 2007, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[41]  D. Finfgeld,et al.  Metasynthesis: The State of the Art—So Far , 2003, Qualitative health research.

[42]  J. Austoker,et al.  Synthesising quantitative and qualitative research in evidence-based patient information , 2007, Journal of Epidemiology and Community Health.

[43]  Karin Hannes,et al.  A Comparative Analysis of Three Online Appraisal Instruments’ Ability to Assess Validity in Qualitative Research , 2010, Qualitative health research.

[44]  Pauline Roux,et al.  Quality of qualitative research in the health sciences: Analysis of the common criteria present in 58 assessment guidelines by expert users. , 2016, Social science & medicine.

[45]  Aron Shlonsky,et al.  Systematic Synthesis of Qualitative Research , 2012 .

[46]  M. Giacomini,et al.  Pregnant women's perceptions of gestational weight gain: A systematic review and meta-synthesis of qualitative research. , 2017, Maternal & child nutrition.

[47]  J. Popay,et al.  Rationale and Standards for the Systematic Review of Qualitative Literature in Health Services Research , 1998, Qualitative health research.

[48]  E. Murphy,et al.  A response to 'Quality in qualitative evaluation: a framework for assessing research evidence' , 2004 .

[49]  Jenny Donovan,et al.  Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. , 2003, Social science & medicine.

[50]  Using Qualitative Assessment and Review Instrument software to synthesise studies on older people's views and experiences of falls prevention. , 2008, International journal of evidence-based healthcare.

[51]  M. Sandelowski,et al.  Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. , 2007, Research in nursing & health.

[52]  D. Moher,et al.  Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement , 2009, BMJ : British Medical Journal.

[53]  Andrew Booth,et al.  Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series , 2018, Implementation Science.

[54]  Margarete Sandelowski,et al.  Finding the findings in qualitative studies. , 2002, Journal of nursing scholarship : an official publication of Sigma Theta Tau International Honor Society of Nursing.

[55]  L. Finlay.  Negotiating the swamp: the opportunity and challenge of reflexivity in research practice , 2002 .