Does a web-based feedback training program result in improved reliability in clinicians' ratings of the Global Assessment of Functioning (GAF) Scale?

The Global Assessment of Functioning (GAF) Scale is used in routine clinical practice and research to estimate symptom and functional severity and longitudinal change. Concerns about poor interrater reliability have been raised, and the present study evaluated the effect of a Web-based GAF training program designed to improve interrater reliability in routine clinical practice. Clinicians rated up to 20 vignettes online, and received deviation scores as immediate feedback (i.e., own scores compared with expert raters) after each rating. Growth curves of absolute SD scores across the vignettes were modeled. A linear mixed effects model, using the clinician's deviation scores from expert raters as the dependent variable, indicated an improvement in reliability during training. Moderation by content of scale (symptoms; functioning), scale range (average; extreme), previous experience with GAF rating, profession, and postgraduate training were assessed. Training reduced deviation scores for inexperienced GAF raters, for individuals in clinical professions other than nursing and medicine, and for individuals with no postgraduate specialization. In addition, training was most beneficial for cases with average severity of symptoms compared with cases with extreme severity. The results support the use of Web-based training with feedback routines as a means to improve the reliability of GAF ratings performed by clinicians in mental health practice. These results especially pertain to clinicians in mental health practice who do not have a masters or doctoral degree.

[1]  D. Cicchetti Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .

[2]  S. Karterud,et al.  The symptom and function dimensions of the Global Assessment of Functioning (GAF) scale. , 2012, Comprehensive psychiatry.

[3]  S. Karterud,et al.  Generalizability studies of the Global Assessment of Functioning-Split version. , 2007, Comprehensive psychiatry.

[4]  L. Adler,et al.  Interrater reliability issues in multicenter trials, Part I: Theoretical concepts and operational procedures used in Department of Veterans Affairs Cooperative Study #394. , 1997, Psychopharmacology bulletin.

[5]  Donald W Smith,et al.  A comparison of psychiatrists' clinical-impression-based and social workers' computer-generated GAF scores. , 2002, Psychiatric services.

[6]  J. Rey,et al.  Inter-rater reliability of global assessment of functioning in a clinical setting. , 1995, Journal of child psychology and psychiatry, and allied disciplines.

[7]  T. Tjus,et al.  Reliability of a functioning scale (GAF) among psychiatric ward staff , 2010 .

[8]  Heinz Katschnig,et al.  Diagnostic and statistical manual of mental disorders, third edition, revised (DSM-III-R) , 1992 .

[9]  R. Telschow,et al.  The longitudinal use of the Global Assessment Scale in multiple-rater situations , 1990, Community Mental Health Journal.

[10]  J. S. Wiggins,et al.  Personality and Prediction: Principles of Personality Assessment , 1973 .

[11]  D. Cook,et al.  Internet-based learning in the health professions: a meta-analysis. , 2008, JAMA.

[12]  Janet B W Williams,et al.  Diagnostic and Statistical Manual of Mental Disorders , 2013 .

[13]  H. Goldman,et al.  Revising axis V for DSM-IV: a review of measures of social functioning. , 1992, The American journal of psychiatry.

[14]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[15]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[16]  George Mendelson,et al.  Book Reviews , 1995 .

[17]  L. Mccullough,et al.  THE RELIABILITY OF THE ACHIEVEMENT OF THERAPEUTIC OBJECTIVES SCALE (ATOS): A RESEARCH AND TEACHING TOOL FOR PSYCHOTHERAPY , 2003 .

[18]  S. Friis,et al.  Are GAF scores reliable in routine clinical use? , 2007, Acta psychiatrica Scandinavica.

[19]  H. Garb,et al.  Clinical judgment, clinical training, and professional experience. , 1989, Psychological bulletin.

[20]  T. Stiles,et al.  The Achievement of Therapeutic Objectives Scale: interrater reliability and sensitivity to change in short-term dynamic psychotherapy and cognitive therapy. , 2011, Psychological assessment.

[21]  S. Friis,et al.  Day treatment of patients with personality disorders: experiences from a Norwegian treatment research network. , 2003, Journal of personality disorders.

[22]  G. Dunbar,et al.  The Mini International Neuropsychiatric Interview (MINI). A short diagnostic structured interview: reliability and validity according to the CIDI , 1997, European Psychiatry.

[23]  Christopher D. Webster,et al.  Assessing risk of violence to others. , 1997 .

[24]  B. Armelius,et al.  Reliability of global assessment of functioning ratings made by clinical psychiatric staff. , 2005, Psychiatric services.

[25]  Genna R. Cohen,et al.  The Meta-Analysis of Clinical Judgment Project , 2009 .

[26]  Eric S. Holmboe,et al.  Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. , 2009, JAMA.

[27]  Factor analysis of the Achievement of Therapeutic Objectives Scale (ATOS) in short-term dynamic psychotherapy and cognitive therapy. , 2014, Psychological assessment.

[28]  F. Zitman,et al.  Reliability and validity of the Global Assessment of Functioning Scale in clinical outpatients with depressive disorders. , 2012, Journal of evaluation in clinical practice.

[29]  S. Ackerman,et al.  Reliability and validity of DSM-IV axis V. , 2000, The American journal of psychiatry.