Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT)

RATIONALE, AIMS, AND OBJECTIVES Systematic reviews combining qualitative, quantitative, and/or mixed methods studies are increasingly popular because of their potential for addressing complex interventions and phenomena, specifically for assessing and improving clinical practice. A major challenge encountered with this type of review is the appraisal of the quality of individual studies given the heterogeneity of the study designs. The Mixed Methods Appraisal Tool (MMAT) was developed to help overcome this challenge. The aim of this study was to explore the usefulness of the MMAT by seeking the views and experiences of researchers who have used it. METHODS We conducted a qualitative descriptive study using semistructured interviews with MMAT users. A purposeful sample was drawn from the researchers who had previously contacted the developer of the MMAT, and those who have published a systematic review for which they had used the MMAT. All interviews were transcribed verbatim and analyzed by 2 coders using thematic analysis. RESULTS Twenty participants from 8 countries were interviewed. Thirteen themes were identified and grouped into the 2 dimensions of usefulness, ie, utility and usability. The themes related to utility concerned the coverage, completeness, flexibility, and other utilities of the tool. Those regarding usability were related to the learnability, efficiency, satisfaction, and errors that could be made due to difficulties understanding or selecting the items to appraise. CONCLUSIONS On the basis of the results of this study, we make several recommendations for improving the MMAT. This will contribute to greater usefulness of the MMAT.

[1]  R. Whittemore,et al.  The integrative review: updated methodology. , 2005, Journal of advanced nursing.

[2]  M. Schmuckler.  What Is Ecological Validity? A Dimensional Analysis. , 2001, Infancy : the official journal of the International Society on Infant Studies.

[3]  Soo Young Kim,et al.  A newly developed tool for classifying study designs in systematic reviews of interventions and exposures showed substantial reliability and validity. , 2016, Journal of clinical epidemiology.

[4]  Iveta Simera,et al.  A proposed framework for developing quality assessment tools , 2017, Systematic Reviews.

[5]  Michael Crowe,et al.  A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed. , 2011, Journal of clinical epidemiology.

[6]  Amanda Burls.  What Is Critical Appraisal , 2015 .

[7]  F. Song,et al.  Evaluating non-randomised intervention studies. , 2003, Health technology assessment.

[8]  M. Neergaard,et al.  Qualitative Description – the poor cousin of health research ? , 2009 .

[9]  Patrick Onghena,et al.  Critical Appraisal of Mixed Methods Studies , 2013 .

[10]  D. Moher,et al.  Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network , 2010, BMC medicine.

[11]  Patrick Onghena,et al.  Mixed methods research synthesis: definition, framework, and potential , 2013 .

[12]  Dominique Boels,et al.  Critical appraisal of mixed methods research studies in a systematic scoping review on plural policing: assessing the impact of excluding inadequately reported studies by means of a sensitivity analysis , 2017 .

[13]  Helen Roberts,et al.  How to Appraise the Studies: An Introduction to Assessing Study Quality , 2008 .

[14]  B. Djulbegovic,et al.  Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols. , 2012, Journal of clinical epidemiology.

[15]  E. Faerstein,et al.  A DICTIONARY OF EPIDEMIOLOGY , 2016 .

[16]  C. Faggion.  Risk of bias assessment should not go beyond reporting assessment. , 2016, Journal of clinical epidemiology.

[17]  J. Fereday,et al.  Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development , 2006 .

[18]  Patrick Onghena,et al.  Using Mixed Methods Research Synthesis for Literature Reviews , 2016 .

[19]  Margarete Sandelowski,et al.  What's in a name? Qualitative description revisited. , 2010, Research in nursing & health.

[20]  A. Booth,et al.  Should We Exclude Inadequately Reported Studies From Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews , 2012, Qualitative health research.

[21]  Pauline Roux,et al.  Quality of qualitative research in the health sciences: Analysis of the common criteria present in 58 assessment guidelines by expert users. , 2016, Social science & medicine.

[22]  Pierre Pluye,et al.  Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. , 2014, Annual review of public health.

[23]  M. Sandelowski,et al.  Reading Qualitative Studies , 2002 .

[24]  Alex J. Sutton,et al.  Integrative approaches to qualitative and quantitative evidence , 2004 .

[25]  S. Haynes,et al.  Content validity in psychological assessment: A functional approach to concepts and methods. , 1995 .

[26]  M. Sandelowski.  Focus on Research Methods Whatever Happened to Qualitative Description? , 2022 .

[27]  Raheleh,et al.  Title External Validity and Model Validity : A Conceptual Approach for Systematic Review Methodology Permalink , 2014 .

[28]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.

[29]  J. Grimshaw,et al.  What is an adequate sample size? Operationalising data saturation for theory-based interview studies , 2010, Psychology & health.

[30]  J. Higgins,et al.  Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. , 2007, International journal of epidemiology.

[31]  R. Khorsan,et al.  How to Assess the External Validity and Model Validity of Therapeutic Trials: A Conceptual Approach to Systematic Review Methodology , 2014, Evidence-based complementary and alternative medicine : eCAM.

[32]  P. Pluye,et al.  Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence , 2017, Systematic Reviews.

[33]  Angela Harden,et al.  Quality and relevance appraisal , 2012 .

[34]  P. Katrak,et al.  A systematic review of the content of critical appraisal tools , 2004, BMC medical research methodology.

[35]  G. Borglin,et al.  Including systematic reviews in PhD programmes and candidatures in nursing - 'Hobson's choice'? , 2014, Nurse education in practice.

[36]  P. Pluye,et al.  A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. , 2009, International journal of nursing studies.

[37]  P. Pluye,et al.  Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. , 2012, International journal of nursing studies.

[38]  P. Pluye,et al.  Systematic mixed studies reviews: updating results on the reliability and efficiency of the Mixed Methods Appraisal Tool. , 2015, International journal of nursing studies.

[39]  S. Masterson,et al.  Factors that motivate individuals to volunteer to be dispatched as first responders in the event of a medical emergency: A systematic review protocol. , 2019, HRB open research.