A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies

The practice of mixed-methods research has increased considerably over the last 10 years. While these studies have been criticized for violating quantitative and qualitative paradigmatic assumptions, the methodological quality of mixed-method studies has not been addressed. The purpose of this paper is to identify criteria to critically appraise the quality of mixed-method studies in the health literature. Criteria for critically appraising quantitative and qualitative studies were generated from a review of the literature. These criteria were organized according to a cross-paradigm framework. We recommend that these criteria be applied to a sample of mixed-method studies which are judged to be exemplary. With the consultation of critical appraisal experts and experienced qualitative, quantitative, and mixed-method researchers, further efforts are required to revise and prioritize the criteria according to importance.

[1]  R. Yin.  Enhancing the quality of case studies in health services research. , 1999, Health services research.

[2]  Thomas A. Schwandt,et al.  Judging interpretations: But is it rigorous? trustworthiness and authenticity in naturalistic evaluation , 2007 .

[3]  Trisha Greenhalgh,et al.  How to read a paper: Papers that go beyond numbers (qualitative research) , 1997 .

[4]  J. Secker,et al.  Qualitative methods in health promotion research: some criteria for quality , 1995 .

[5]  Valerie J. Caracelli,et al.  Advances in mixed-method evaluation : the challenges and benefits of integrating diverse paradigms , 1997 .

[6]  Jennifer Caroline Greene,et al.  Data Analysis Strategies for Mixed-Method Evaluation Designs , 1993 .

[7]  T. Greenhalgh,et al.  Papers that go beyond numbers (qualitative research) , 1997, BMJ.

[8]  Gordon H. Guyatt,et al.  Users' Guides to the Medical Literature: I. How to Get Started , 1993 .

[9]  R. Elliott,et al.  Evolving guidelines for publication of qualitative research studies in psychology and related fields. , 1999, The British journal of clinical psychology.

[10]  How to read clinical journals: III. To learn the clinical course and prognosis of disease. , 1981, Canadian Medical Association journal.

[11]  J. Morse.  Evaluating Qualitative Research , 1991 .

[12]  E. Guba.  The alternative paradigm dialog. , 1990 .

[13]  Sharon F. Rallis,et al.  Qualitative and quantitative inquiries are not incompatible: A call for a new partnership , 1994 .

[14]  How to read clinical journals: II. To learn about a diagnostic test. , 1981, Canadian Medical Association journal.

[15]  M J Campbell,et al.  Use of check lists in assessing the statistical content of medical studies. , 1986, British medical journal.

[16]  P. Hodkinson,et al.  Can There Be Criteria for Selecting Research Criteria?—A Hermeneutical Analysis of an Inescapable Dilemma , 1998 .

[17]  F G Fowkes,et al.  Critical appraisal of published research: introductory guidelines. , 1991, BMJ.

[18]  E. Guba,et al.  Competing paradigms in qualitative research. , 1994 .

[19]  Abbas Tashakkori,et al.  Mixed Methodology: Combining Qualitative and Quantitative Approaches , 1998 .

[20]  J. Popay,et al.  Rationale and Standards for the Systematic Review of Qualitative Literature in Health Services Research , 1998, Qualitative health research.

[21]  David L. Altheide,et al.  Criteria for assessing interpretive validity in qualitative research. , 1994 .

[22]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[23]  Lois-ellin Datta,et al.  Multimethod Evaluations: Using Case Studies Together with Other Methods , 1997 .

[24]  P Tugwell,et al.  Users' guides to the medical literature. V. How to use an article about prognosis. Evidence-Based Medicine Working Group. , 1994, JAMA.

[25]  J. Creswell.  Qualitative inquiry and research design: choosing among five traditions. , 1998 .

[26]  T. Greenhalgh.  How to read a paper: Statistics for the non-statistician. II: 'Significant' relations and their pitfalls , 1997 .

[27]  J. Smith.  Quantitative Versus Qualitative Research: An Attempt to Clarify the Issue , 1983 .

[28]  R. Foster,et al.  Addressing Epistemologic and Practical Issues in Multimethod Research: A Procedure for Conceptual Triangulation , 1997, ANS. Advances in nursing science.

[29]  M Ford-Gilboe,et al.  Stories and numbers: coexistence without compromise. , 1995, ANS. Advances in nursing science.

[30]  T. Koch,et al.  Establishing rigour in qualitative research: the decision trail. , 1994, Journal of advanced nursing.

[31]  Gordon H. Guyatt,et al.  How to Use an Article About Therapy or Prevention , 1995 .

[32]  J N Hagemaster,et al.  Ten criteria for evaluating qualitative research proposals. , 1987, The Journal of nursing education.

[33]  Ernest R. House,et al.  Integrating the quantitative and qualitative , 1994 .

[34]  G H Guyatt,et al.  Users' guides to the medical literature. I. How to get started. The Evidence-Based Medicine Working Group. , 1993, JAMA.

[35]  A. Sparkes.  Myth 94: Qualitative Health Researchers will Agree about Validity , 2001, Qualitative health research.

[36]  How to read clinical journals: V: To distinguish useful from useless or even harmful therapy. , 1981, Canadian Medical Association journal.

[37]  G H Guyatt,et al.  Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. , 1994, JAMA.

[38]  H. S. Wilson.  Qualitative Studies: From Observations to Explanations , 1985, The Journal of nursing administration.

[39]  Anton J. Kuzel,et al.  Standards of trustworthiness for qualitative studies in primary care. , 1991 .

[40]  K. Brazil,et al.  Revisiting the Quantitative-Qualitative Debate: Implications for Mixed-Methods Research , 2002, Quality & quantity.

[41]  K. Devers.  How will we know "good" qualitative research when we see it? Beginning the dialogue in health services research. , 1999, Health services research.

[42]  R. Berk.  The Construction of Rating Instruments for Faculty Evaluation: A Review of Methodological Issues. , 1979 .

[43]  Valerie J. Caracelli,et al.  Mixed-method evaluation: Developing quality criteria through concept mapping: Mixed-Method Collaboration , 1994 .

[44]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[45]  P. Berger,et al.  The Social Construction of Reality , 1966 .

[46]  A J Reid,et al.  What we want: qualitative research. Promising frontier for family medicine. , 1996, Canadian family physician Medecin de famille canadien.

[47]  D. Sackett,et al.  Users' guides to the medical literature. IV. How to use an article about harm. Evidence-Based Medicine Working Group. , 1994, JAMA.

[48]  C Forchuk,et al.  How to critique qualitative research articles. , 1993, The Canadian journal of nursing research = Revue canadienne de recherche en sciences infirmieres.

[49]  N. Denzin,et al.  Handbook of Qualitative Research , 1994 .

[50]  M. Patton,et al.  Enhancing the quality and credibility of qualitative analysis. , 1999, Health services research.

[51]  J. Droitcour.  Cross-Design Synthesis: Concept and Application , 1997 .

[52]  N. Burns.  Standards for Qualitative Research , 1989, Nursing science quarterly.

[53]  J. Marsden,et al.  Assessing methodological quality of published papers , 1998, BMJ.

[54]  D L Morgan,et al.  Practical Strategies for Combining Qualitative and Quantitative Methods: Applications to Health Research , 1998, Qualitative health research.

[55]  Gordon H. Guyatt,et al.  Users' Guides to the Medical Literature: V. How to Use an Article About Prognosis , 1994 .

[56]  L. Stewin,et al.  Reliability and validity: misnomers for qualitative research. , 1988, The Canadian journal of nursing research = Revue canadienne de recherche en sciences infirmieres.

[57]  A. Casebeer,et al.  Combining qualitative and quantitative research methods: considering the possibilities for enhancing the study of chronic diseases. , 1997, Chronic diseases in Canada.