Instruments for assessing the quality of drug studies published in the medical literature.

OBJECTIVES To develop valid and reliable instruments to assess the methodologic quality and clinical relevance of drug studies. DESIGN We developed an instrument to assess the methodologic quality of articles reporting clinical research and an instrument to measure nonmethodologic measures of quality, such as clinical relevance, generalizability, and adherence to ethical standards. Each instrument was pretested by seven independent, masked reviewers and modified based on interrater agreement and content validity of individual items. We determined correlational validity of the final methodologic quality instrument by comparing quality scores assigned to 10 articles by means of our instrument and a previously published one. PARTICIPANTS Clinical drug studies published in symposium proceedings and peer reviewed biomedical literature. MAIN OUTCOME MEASURES Interrater reliability of overall quality scores, measured by intraclass correlation (r) and Kendall's coefficient of concordance (W), and interrater reliability of individual items, by percentage agreement. MAIN RESULTS The interrater reliability of the pretest methodologic quality instrument was high (r = .89 [95% confidence interval, .73 to .96]; W = 0.64). Correlational validity of the final instrument was suggested by the high degree of concordance with another previously published one (W = 0.74). The interrater reliability of the pretest clinical relevance instrument was moderate (r = .41 [95% confidence interval, .18 to .64]; W = 0.47). Reviewers confirmed the content validity of both instruments. CONCLUSIONS The two instruments we developed, one measuring methodologic quality and one measuring clinical relevance of articles reporting clinical research, are reliable, valid, and applicable to a variety of research designs.

[1]  T C Chalmers,et al.  A method for assessing the quality of a randomized control trial. , 1981, Controlled clinical trials.

[2]  V. Goel,et al.  Perioperative parenteral nutrition: a meta-analysis. , 1987, Annals of internal medicine.

[3]  W. A. Scott,et al.  Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. , 1974 .

[4]  G H Guyatt,et al.  USERS' GUIDES TO THE MEDICAL LITERATURE. II: HOW TO USE AN ARTICLE ABOUT THERAPY OR PREVENTION A. ARE THE RESULTS OF THE STUDY VALID ? , 1993 .

[5]  U. P. S. T. Force,et al.  Guide to clinical preventive services : an assessment of the effectiveness of 169 interventions : report of the U.S. Preventive Services Task Force , 1989 .

[6]  Wirt M. Wolff,et al.  A study of criteria for journal manuscripts. , 1970 .

[7]  L Abenhaim,et al.  Links between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking. , 1990, Clinical and investigative medicine. Medecine clinique et experimentale.

[8]  E. A. Haggard,et al.  Intraclass Correlation and the Analysis of Variance , 1958 .

[9]  J. Dolovich,et al.  The need for extended treatment of anaphylaxis. , 1980, Canadian Medical Association journal.

[10]  Gordon H. Guyatt,et al.  Users' Guides to the Medical Literature: II. How to Use an Article About Therapy or Prevention B. What Were the Results and Will They Help Me in Caring for My Patients? , 1994 .

[11]  K A L'Abbé,et al.  Meta-analysis in clinical research. , 1987, Annals of internal medicine.

[12]  E. Hemminki.  Quality of Clinical Trials — A Concern of Three Decades , 1982, Methods of Information in Medicine.

[13]  A Liberati,et al.  A quality assessment of randomized control trials of primary treatment of breast cancer. , 1986, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[14]  G. S. Cooper,et al.  An analysis of the quality of research reports in theJournal of General Internal Medicine , 1989, Journal of general internal medicine.

[15]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[16]  C D Naylor,et al.  Incorporating variations in the quality of individual randomized trials into meta-analysis. , 1992, Journal of clinical epidemiology.

[17]  Frederick Mosteller,et al.  Reporting on methods in clinical trials. , 1982, The New England journal of medicine.

[18]  E Andrew,et al.  Publications on clinical trials with X-ray contrast media: differences in quality between journals and decades. , 1990, European journal of radiology.

[19]  Paul McReynolds,et al.  Reliability of ratings of research papers. , 1971 .

[20]  E Andrew,et al.  Method for Assessment of the Reporting Standard of Clinical Trials with Roentgen Contrast Media , 1984, Acta radiologica: diagnosis.