Addressing quality in mixed methods research: a review and recommendations for a future agenda

Quality is one of the most debated topics in the recent history of mixed methods research. A growing number of authors are currently discussing how the quality of mixed methods research should be conceptualised and operationalised, with the ultimate aim of promoting well designed and properly implemented mixed methods studies. These authors argue that mixed methods research has a number of unique features with respect to monomethod research and, as such, should be appraised according to its own set of quality criteria. Based on a systematic search of the literature, this review (1) examines the features and trends of the literature on the quality of mixed methods research published until February 2016; (2) provides a metasummary of the most prevalent quality criteria suggested in this literature; and (3) proposes a number of recommendations for future discussion and research on this topic. The review concludes that (1) publications on the quality of mixed methods research are increasingly more prevalent and elaborated; (2) a shared set of core quality criteria for appraising mixed methods research can be identified across publications; and (3) future work on this topic should focus on increasing the number of empirical publications on quality, achieving greater consistency in quality terminology, and reaching an agreement on core quality criteria.

[1]  Antigoni Papadimitriou,et al.  Addressing Challenges of Conducting Quality Mixed Methods Studies in Higher Education , 2014 .

[2]  A. Bryman June 1989 and beyond: Julia Brannen’s contribution to mixed methods research , 2014 .

[3]  Lili-Anne Kihn,et al.  Threats to validity and reliability in mixed methods accounting research , 2011 .

[4]  Alan Bryman,et al.  Paradigm Peace and the Implications for Quality , 2006 .

[5]  Genevieve Canales Transformative, Mixed Methods Checklist for Psychological Research With Mexican Americans , 2013 .

[6]  A. Tashakkori OVERVIEW OF CONTEMPORARY ISSUES IN MIXED METHODS RESEARCH , 2010 .

[7]  Nancy L. Leech,et al.  A typology of mixed methods research designs , 2009 .

[8]  N. Ivankova,et al.  Mixed Methods Research: A Guide to the Field , 2015 .

[9]  Nancy L. Leech,et al.  Writing Mixed Research Reports , 2012 .

[10]  V. Reed,et al.  Using mixed methods research in medical education: basic guidelines for researchers , 2009, Medical education.

[11]  Sharlene Hesse-Biber,et al.  Coming at Things Differently , 2013 .

[12]  Jennifer Caroline Greene,et al.  Mixed Methods in Social Inquiry , 2007 .

[13]  Martyn Hammersley,et al.  The issue of quality in qualitative research , 2007 .

[14]  Anthony J. Onwuegbuzie,et al.  Editors’ Afterword , 2016 .

[15]  Donna M. Mertens Publishing Mixed Methods Research , 2011 .

[16]  A. O’Cathain,et al.  The Quality of Mixed Methods Studies in Health Services Research , 2008, Journal of health services research & policy.

[17]  A. Onwuegbuzie,et al.  Methodological reporting in qualitative, quantitative, and mixed methods health services research articles. , 2012, Health services research.

[18]  José F. Molina-Azorín,et al.  The Future of Mixed Methods: A Five Year Projection to 2020 , 2016 .

[19]  John W. Creswell,et al.  Editorial: Developing Publishable Mixed Methods Manuscripts , 2007 .

[20]  Marcella Nunez-Smith,et al.  Mixed Methods in Health Sciences Research: A Practical Primer , 2014 .

[21]  Nataliya V. Ivankova,et al.  Mixed Methods Applications in Action Research: From Methods to Community Action , 2014 .

[22]  Kathleen M. T. Collins,et al.  Securing a Place at the Table , 2012 .

[23]  J. Creswell Revisiting Mixed Methods and Advancing Scientific Practices , 2015 .

[24]  Kathleen M. T. Collins,et al.  A Mixed Methods Investigation of Mixed Methods Sampling Designs in Social and Health Science Research , 2007 .

[25]  C. Seale,et al.  Quality in Qualitative Research , 1999 .

[26]  Abbas Tashakkori,et al.  Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences , 2008 .

[27]  C. Barber,et al.  Twelve tips for getting started using mixed methods in medical education research , 2013, Medical teacher.

[28]  Alan Bryman,et al.  Quality Criteria for Quantitative, Qualitative and Mixed Methods Research: A View from Social Policy , 2008 .

[29]  P. Pluye,et al.  A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. , 2009, International journal of nursing studies.

[30]  Nancy L. Leech,et al.  Validity: Mixed Methods , 2012 .

[31]  J. Gallo,et al.  Funding and Publishing Integrated Studies: Writing Effective Mixed Methods Manuscripts and Grant Proposals , 2010 .

[32]  A. O’Cathain Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework , 2010 .

[33]  E. Halcomb,et al.  Mixed methods research. , 2015, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[34]  A. Onwuegbuzie,et al.  The Validity Issue in Mixed Research , 2006 .

[35]  Vicki L. Plano Clark,et al.  Mixed Methods Approaches in Family Science Research , 2008 .

[36]  Alan Bryman,et al.  The Research Question in Social Research: What is its Role? , 2007 .

[37]  Kevin Brazil,et al.  A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies , 2004, Quality & quantity.

[38]  S. Leatherdale,et al.  Searching for rigour in the reporting of mixed methods population health research: a methodological review. , 2015, Health education research.

[39]  Timothy J. Burrows,et al.  A Preliminary Rubric Design to Evaluate Mixed Methods Research. , 2013 .

[40]  Janice M. Morse,et al.  Mixed Method Design: Principles and Procedures , 2009 .

[41]  L. Giddings,et al.  From Rigour to Trustworthiness: Validating Mixed Methods , 2009 .

[42]  A. Bryman Integrating quantitative and qualitative research: how is it done? , 2006 .

[43]  Abbas Tashakkori,et al.  Epilogue: Current Developments and Emerging Trends in Integrated Research Methodology , 2010 .

[44]  John W. Creswell,et al.  A Concise Introduction to Mixed Methods Research , 2014 .

[45]  Nataliya V. Ivankova,et al.  Implementing Quality Criteria in Designing and Conducting a Sequential QUAN → QUAL Mixed Methods Study of Student Engagement With Learning Applied Research Methods Online , 2014 .

[46]  Margarete Sandelowski,et al.  Handbook for Synthesizing Qualitative Research , 2006 .

[47]  John W. Creswell,et al.  Designing and Conducting Mixed Methods Research , 2006 .

[48]  Nancy L. Leech Interviews with the Early Developers of Mixed Methods Research1 , 2010 .

[49]  Valerie J. Caracelli,et al.  Mixed-method evaluation: Developing quality criteria through concept mapping: Mixed-Method Collaboration , 1994 .

[50]  Kathleen M. T. Collins,et al.  Assessing legitimation in mixed research: a new framework , 2011 .

[51]  I. Newman,et al.  Mixed Methods Research , 2008 .

[52]  A. Onwuegbuzie,et al.  Toward a Definition of Mixed Methods Research , 2007 .

[53]  N. Leech,et al.  Toward a Unified Validation Framework in Mixed Methods Research , 2007 .

[54]  Nataliya V. Ivankova,et al.  Emerging Trends in the Utilization of Integrated Designs in the Social, Behavioral, and Health Sciences , 2010 .

[55]  Anthony J. Onwuegbuzie,et al.  Improving the Quality of Mixed Research Reports in the Field of Human Resource Development and Beyond: A Call for Rigor as an Ethical Practice , 2014 .

[56]  Patrick Onghena,et al.  Critical Appraisal of Mixed Methods Studies , 2013 .

[57]  J. Willis Foundations of Qualitative Research: Interpretive and Critical Approaches , 2007 .

[58]  Viswanath Venkatesh,et al.  Bridging the Qualitative-Quantitative Divide: Guidelines for Conducting Mixed Methods Research in Information Systems , 2013, MIS Q..

[59]  S. Closs,et al.  Mixed‐methods research in pharmacy practice: recommendations for quality reporting (part 2) , 2014, The International journal of pharmacy practice.

[60]  John H. Hitchcock,et al.  An Inclusive Framework for Conceptualizing Mixed Methods Design Typologies: Moving Toward Fully Integrated Synergistic Research Models , 2010 .

[61]  J. Creswell,et al.  Best Practices for Mixed Methods Research in Health Sciences: (566732013-001) , 2011 .

[62]  José F. Molina-Azorίn The Use and Added Value of Mixed Methods in Management Research , 2011 .