Research methods: managing primary study quality in meta-analyses.

Meta-analyses synthesize multiple primary studies and identify patterns of relationships. Differences in primary study methodological quality must be addressed for meta-analysis to produce meaningful results. No single standard exists for addressing these quality variations. Quality measurement scales are fraught with development and application problems. Several strategies have been proposed to address quality. Researchers can set minimum levels for inclusion or require that certain quality attributes be present. An inclusive method is to weight effect sizes by quality scores. This allows the inclusion of diverse studies but relies on questionable quality measures. By considering quality an empirical question, meta-analysts can examine associations between quality and effect sizes and thus preserve the purpose of meta-analysis to systematically examine data. Researchers increasingly are combining strategies to overcome the limitations of using a single approach. Future work to develop valid measures of primary study quality dimensions will improve the ability of meta-analysis to inform research and nursing practice.

[1]  S Greenland,et al.  Invited commentary: a critical look at some popular meta-analytic methods. , 1994, American journal of epidemiology.

[2]  I. Chalmers,et al.  The Landscape and Lexicon of Blinding in Randomized Trials , 2002, Annals of Internal Medicine.

[3]  I Olkin,et al.  Meta-analysis of randomized controlled trials. A concern for standards;. , 1995, JAMA.

[4]  Douglas G Altman,et al.  Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta‐epidemiological’ research , 2002, Statistics in medicine.

[5]  K. Khan,et al.  When small degrees of bias in randomized trials can mislead clinical decisions: An example of individualizing preventive treatment of upper gastrointestinal bleeding , 2002, Critical care medicine.

[6]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[7]  Paul Knipschild,et al.  Systematic Reviews on the Basis of Methodological Criteria , 1997 .

[8]  A D Oxman,et al.  Systematic Reviews: Checklists for review articles , 1994 .

[9]  A R Jadad,et al.  Assessing the Quality of Randomized Controlled Trials: Current Issues and Future Directions , 1996, International Journal of Technology Assessment in Health Care.

[10]  R. Hyman Quasi-Experimentation: Design and Analysis Issues for Field Settings (Book) , 1982 .

[11]  A. Laupacis,et al.  WHAT SHOULD BE INCLUDED IN META-ANALYSES? , 2000, International Journal of Technology Assessment in Health Care.

[12]  D Tritchler Modelling study quality in meta-analysis. , 1999, Statistics in medicine.

[13]  Bias in treatment assignment in controlled clinical trials. , 1984, The New England journal of medicine.

[14]  D. Cook,et al.  Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. , 1999, Health technology assessment.

[15]  D Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Annals of internal medicine.

[16]  S. Brown Meta-analysis of diabetes patient education research: variations in intervention effects across studies. , 1992, Research in nursing & health.

[17]  C. Eccleston,et al.  Systematic review and meta-analysis of randomized controlled trials of cognitive behaviour therapy and behaviour therapy for chronic pain in adults, excluding headache , 1999, Pain.

[18]  J Gabbay,et al.  'Early warning systems' for identifying new healthcare technologies. , 1999, Health technology assessment.

[19]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[20]  L. Bouter,et al.  Steroid injections for shoulder disorders: a systematic review of randomized clinical trials. , 1996, The British journal of general practice : the journal of the Royal College of General Practitioners.

[21]  D. Sharpe,et al.  Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away. , 1997, Clinical psychology review.

[22]  Paul M. Wortman,et al.  Judging research quality. , 1994 .

[23]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[24]  Ethan M Balk,et al.  Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. , 2002, JAMA.

[25]  M. Suarez‐Almazor,et al.  The efficacy of folic acid and folinic acid in reducing methotrexate gastrointestinal toxicity in rheumatoid arthritis. A metaanalysis of randomized controlled trials. , 1998, The Journal of rheumatology.

[26]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[27]  A. Jadad,et al.  The importance of quality of primary studies in producing unbiased systematic reviews. , 1996, Archives of internal medicine.

[28]  C D Naylor,et al.  Incorporating variations in the quality of individual randomized trials into meta-analysis. , 1992, Journal of clinical epidemiology.

[29]  Jeffrey C. Valentine,et al.  The Department of Education’s What Works Clearinghouse Study Design and Implementation Assessment Device: Version 1.0 , 2003 .

[30]  A. Vickers,et al.  INCORPORATING DATA FROM DISSERTATIONS IN SYSTEMATIC REVIEWS , 2000, International Journal of Technology Assessment in Health Care.

[31]  Christian Gluud,et al.  Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses , 2001, Annals of Internal Medicine.

[32]  Douglas G Altman,et al.  Systematic reviews in health care: Assessing the quality of controlled clinical trials. , 2001, BMJ.

[33]  A R Jadad,et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary? , 1996, Controlled clinical trials.

[34]  A R Jadad,et al.  Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. , 1995, Controlled clinical trials.

[35]  K. Lohr,et al.  Assessing "best evidence": issues in grading the quality of studies for systematic reviews. , 1999, The Joint Commission journal on quality improvement.

[36]  N. Black,et al.  The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. , 1998, Journal of epidemiology and community health.

[37]  P. Raina,et al.  Assessing the Methodological Quality of Nonrandomized Intervention Studies , 2003, Western journal of nursing research.

[38]  Cheryl Tatano Beck Facilitating the work of a meta-analyst. , 1999, Research in nursing & health.

[39]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.

[40]  J D Emerson,et al.  An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. , 1990, Controlled clinical trials.

[41]  On the Social Psychology of Using Research: The Case of Desegregation and Black Achievement. , 1984 .

[42]  A D Oxman,et al.  The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials , 1998, BMJ.

[43]  W B Jonas,et al.  Impact of study quality on outcome in placebo-controlled trials of homeopathy. , 1999, Journal of clinical epidemiology.

[44]  A Liberati,et al.  A quality assessment of randomized control trials of primary treatment of breast cancer. , 1986, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[45]  R. Fletcher,et al.  The effects of blinding on the quality of peer review. A randomized trial. , 1990, JAMA.

[46]  K. Seers,et al.  Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. , 1997, Journal of advanced nursing.

[47]  F. Mosteller,et al.  How study design affects outcomes in comparisons of therapy. I: Medical. , 1989, Statistics in medicine.

[48]  L. Bouter,et al.  The relationship between methodological quality and conclusions in reviews of spinal manipulation. , 1995, JAMA.

[49]  T C Chalmers,et al.  A method for assessing the quality of a randomized control trial. , 1981, Controlled clinical trials.

[50]  Jeffrey C. Valentine,et al.  Interventions to increase physical activity among aging adults: A meta-analysis , 2002, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[51]  V. Conn,et al.  Meta-analysis and public policy: opportunity for nursing impact. , 1996, Nursing outlook.