Reliability and validity in binary ratings: areas of common misunderstanding in diagnosis and symptom ratings.

Confusion may exist between the reliability of a binary rating (for example, schizophrenia versus not-schizophrenia) and its implications for validity. High reliability does not guarantee validity, but paradoxically, low reliability does not imply poor validity in all contexts. Changes in the base rate or in experimental design may indicate high validity even when the reliability was thought to be low. Attempts to improve the psychiatric nomenclature by increasing only reliability run the risk of the "attenuation paradox" where further increases in reliability will make the ratings less valid. Finally, the assumption of random error in making diagnoses does not always hold, so that statistical analyses must be adjusted accordingly. New statistical methods are needed to index only false-positive or false-negative rates in order to quantify the error that will reduce some validity coefficients.

[1]  R. Priest.  Measurement and Classification of Psychiatric Symptoms , 1975 .

[2]  J. Loevinger,et al.  The attenuation paradox in test theory. , 1954, Psychological bulletin.

[3]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[4]  M. Lorant.  WHO International Pilot Study of Schizophrenia , 1967, Psychological Medicine.

[5]  J. Fleiss.  Measuring agreement between two judges on the presence or absence of a trait. , 1975, Biometrics.

[6]  J. Fleiss,et al.  A Re-analysis of the Reliability of Psychiatric Diagnosis , 1974, British Journal of Psychiatry.

[7]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[8]  James Shields,et al.  Etiology of Psychosis. (Book Reviews: Schizophrenia and Genetics. A Twin Study Vantage Point) , 1972 .

[9]  J. Feighner,et al.  Diagnostic criteria for use in psychiatric research. , 1972, Archives of general psychiatry.

[10]  K. Krippendorff.  Bivariate Agreement Coefficients for Reliability of Data , 1970 .

[11]  J. Shields,et al.  Cross-national diagnosis of schizophrenia in twins. The heritability and specificity of schizophrenia. , 1972, Archives of general psychiatry.

[12]  J. Fleiss,et al.  Constraints on the validity of computer diagnosis. , 1974, Archives of general psychiatry.

[13]  J. S. Wiggins,et al.  Personality and Prediction: Principles of Personality Assessment , 1973 .

[14]  P. Meehl,et al.  Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. , 1955, Psychological bulletin.

[15]  S. Guze,et al.  A Family Study of Hysteria , 1963 .

[16]  M. Fischer,et al.  DEVELOPMENT AND VALIDITY OF A COMPUTERIZED METHOD FOR DIAGNOSES OF FUNCTIONAL PSYCHOSES (DIAX) , 1974 .

[17]  E. Robins,et al.  Clinical criteria for psychiatric diagnosis and DSM-III. , 1975, The American journal of psychiatry.

[18]  A E Maxwell,et al.  Coefficients of Agreement Between Observers and Their Interpretation , 1977, British Journal of Psychiatry.