Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project

validated criteria to assess the quality of guidelines. Objective: To develop and validate an international instrument for assessing the quality of the process and reporting of clinical practice guideline development. Methods: The instrument was developed through a multi-staged process of item generation, selection and scaling, field testing, and refinement procedures. 100 guidelines selected from 11 participating countries were evaluated independently by 194 appraisers with the instrument. Following refinement the instrument was further field tested on three guidelines per country by a new set of 70 appraisers. Results: The final version of the instrument contained 23 items grouped into six quality domains with a 4 point Likert scale to score each item (scope and purpose, stakeholder involvement, rigour of devel- opment, clarity and presentation, applicability, editorial independence). 95% of appraisers found the instrument useful for assessing guidelines. Reliability was acceptable for most domains (Cronbach's alpha 0.64-0.88). Guidelines produced as part of an established guideline programme had significantly higher scores on editorial independence and, after the publication of a national policy, had significantly higher quality scores on rigour of development (p<0.005). Guidelines with technical documentation had higher scores on that domain (p<0.0001). Conclusions: This is the first time an appraisal instrument for clinical practice guidelines has been developed and tested internationally. The instrument is sensitive to differences in important aspects of guidelines and can be used consistently and easily by a wide range of professionals from different backgrounds. The adoption of common standards should improve the consistency and quality of the reporting of guideline development worldwide and provide a framework to encourage international comparison of clinical practice guidelines.

[1]  H. Toutenburg.  Fleiss, J. L.: Statistical Methods for Rates and Proportions. John Wiley & Sons, New York‐London‐Sydney‐Toronto 1973. XIII, 233 S. , 1974 .

[2]  J. Marks.  An International Overview , 1985 .

[3]  Siep Thomas,et al.  Attributes of clinical guidelines that influence use of guidelines in general practice: observational study , 1998, BMJ.

[4]  J. Grimshaw,et al.  Development and application of a generic methodology to assess the quality of clinical guidelines. , 1999, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[5]  R. Thomson,et al.  Fortnightly Review: How to ensure that guidelines are effective , 1995 .

[6]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[7]  D. Moher,et al.  Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. , 2001, JAMA.

[8]  D Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Annals of internal medicine.

[9]  R. MacCallum,et al.  Sample size in factor analysis. , 1999 .

[10]  J. Grimshaw,et al.  Potential benefits, limitations, and harms of clinical guidelines , 1999, BMJ.

[11]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[12]  J. Grimshaw,et al.  Achieving health gain through clinical guidelines. I: Developing scientifically valid guidelines. , 1993, Quality in health care : QHC.

[13]  D. Altman,et al.  Statistics notes: Cronbach's alpha , 1997 .

[14]  J. Fleiss,et al.  The measurement of interrater agreement , 2004 .

[15]  P. Shekelle,et al.  Clinical guidelines: developing guidelines. , 1999, BMJ.

[16]  Alessandro Liberati,et al.  Practice guidelines developed by specialty societies: the need for a critical appraisal , 2000, The Lancet.

[17]  M. Field,et al.  A Provisional Instrument for Assessing Clinical Practice Guidelines , 1992 .

[18]  Martin Eccles,et al.  Developing Clinical Practice Guidelines , 1998 .

[19]  M. Field,et al.  Guidelines for Clinical Practice: From Development to Use , 1992 .

[20]  M. Mayo-Smith,et al.  Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. , 1999, JAMA.

[21]  R S Hayward,et al.  Users' guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The Evidence-Based Medicine Working Group. , 1995, JAMA.

[22]  P. Littlejohns,et al.  Appraising clinical practice guidelines in England and Wales: the development of a methodologic framework and its application to policy. , 1999, The Joint Commission journal on quality improvement.