Fuzzy kappa for the agreement measure of fuzzy classifications

In this paper, we propose an assessment method of agreement between fuzzy sets, called fuzzy Kappa which is deduced from the concept of Cohen's Kappa statistic. In fuzzy case, the Cohen's Kappa coefficient can be calculated generally by transforming the fuzzy sets into some crisp @a-cut subsets. While the proposed fuzzy Kappa allows to directly evaluate an overall agreement between two fuzzy sets. Hence, it is an efficient agreement measure between a given fuzzy ''ground truth'' or reference and a result of fuzzy classification or fuzzy segmentation. Based on membership function, we define its agreement function and its probability distribution to formulate the deduction of the expectation agreement. So the fuzzy Kappa is calculated from the proportion of the observed agreement and the agreement expected by chance. All the definitions and deductions are detailed in this paper. Both Cohen's Kappa and the fuzzy Kappa are then used to evaluate the agreement between a fuzzy classification of brain tissues on MRI images and its ''ground truth''. A comparison of the two types of Kappa coefficient is carried out and shows the advantage of the fuzzy Kappa and some limitations of Cohen's Kappa in the fuzzy case.

[1]  H. Kraemer,et al.  Extension of the kappa coefficient. , 1980, Biometrics.

[2]  J. Fleiss,et al.  Statistical methods for rates and proportions , 1973 .

[3]  Anthony P. Reeves,et al.  Classification quality assessment for a generalized model-based object identification system , 1989, IEEE Trans. Syst. Man Cybern..

[4]  Jean-Marc Constans,et al.  Segmentation based on information fusion applied to brain tissue on MRI , 2004, IS&T/SPIE Electronic Imaging.

[5]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[6]  H. Lee-Kwang,et al.  Similarity measure between fuzzy sets and between elements , 1994 .

[7]  Li Fan.  Measures of Similarity between Vague Sets , 2001 .

[8]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[9]  Ho-Won Jung,et al.  Evaluating interrater agreement in SPICE-based assessments , 2003, Comput. Stand. Interfaces.

[10]  A. Palmer,et al.  A fuzzy classification technique for predicting species’ distributions: applications using invasive alien plants and indigenous insects , 2004 .

[11]  Bernice W. Polemis.  Nonparametric Statistics for the Behavioral Sciences , 1959 .

[12]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[13]  Alex Hagen,et al.  Fuzzy set approach to assessing similarity of categorical maps , 2003, Int. J. Geogr. Inf. Sci..

[14]  Mohamed-Jalal Fadili,et al.  Fuzzy Markovian Segmentation in Application of Magnetic Resonance Images , 2002, Comput. Vis. Image Underst..

[15]  Mark A. Friedl,et al.  Estimating pixel-scale land cover classification confidence using nonparametric machine learning methods , 2001, IEEE Trans. Geosci. Remote. Sens..

[16]  C. Berry.  The κ Statistic , 1992 .

[17]  John O. Greene,et al.  Cognition and Talk: The Relationship of Semantic Units to Temporal Patterns of Fluency in Spontaneous Speech , 1986 .

[18]  Ronald L. Goldman.  The κ Statistic-Reply , 1992 .

[19]  K. Krippendorff.  Krippendorff, Klaus, Content Analysis: An Introduction to its Methodology . Beverly Hills, CA: Sage, 1980. , 1980 .

[20]  Berry Cc.  The kappa statistic. , 1992 .

[21]  Konstantinos Sirlantzis,et al.  On the relation between dependence and diversity in multiple classifier systems , 2005, International Conference on Information Technology: Coding and Computing (ITCC'05) - Volume II.

[22]  M Ker,et al.  Issues in the use of kappa. , 1991, Investigative radiology.

[23]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[24]  George Hripcsak,et al.  Measuring agreement in medical informatics reliability studies , 2002, J. Biomed. Informatics.

[25]  P. Prescott,et al.  Issues in the Use of Kappa to Estimate Reliability , 1986, Medical care.

[26]  C. Berry,et al.  The kappa statistic. , 1992, JAMA.

[27]  Lorenzo Bruzzone,et al.  Quality assessment of classification and cluster maps without ground truth knowledge , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[28]  L. H. Olsen,et al.  Kappa statistics in the assessment of observer variation: the significance of multiple observers classifying ankle fractures , 2002, Journal of orthopaedic science : official journal of the Japanese Orthopaedic Association.

[29]  S Caeiro,et al.  SADO ESTUARY MANAGEMENT AREAS : HARD VERSUS SOFT CLASSIFICATION MAPS COMPARISON , 2003 .

[30]  Zhang Wen.  Measures of similarity between vague sets , 2004 .

[31]  Alan C. Evans,et al.  BrainWeb: Online Interface to a 3D MRI Simulated Brain Database , 1997 .