Exploring Symmetry of Binary Classification Performance Metrics

Selecting the proper performance metric constitutes a key issue for most classification problems in the field of machine learning. Although the specialized literature has addressed several topics regarding these metrics, their symmetries have yet to be systematically studied. This research focuses on ten metrics based on a binary confusion matrix and their symmetric behaviour is formally defined under all types of transformations. Through simulated experiments, which cover the full range of datasets and classification results, the symmetric behaviour of these metrics is explored by exposing them to hundreds of simple or combined symmetric transformations. Cross-symmetries among the metrics and statistical symmetries are also explored. The results obtained show that, in all cases, three and only three types of symmetries arise: labelling inversion (between positive and negative classes); scoring inversion (concerning good and bad classifiers); and the combination of these two inversions. Additionally, certain metrics have been shown to be independent of the imbalance in the dataset and two cross-symmetries have been identified. The results regarding their symmetries reveal a deeper insight into the behaviour of various performance metrics and offer an indicator to properly interpret their values and a guide for their selection for certain specific applications.

[1]  A. V. Shubnikov,et al.  Symmetry in Science and Art , 1974 .

[2]  Symmetry in mathematics , 1992 .

[3]  Francisco Herrera,et al.  An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics , 2013, Inf. Sci..

[4]  Swagatam Das,et al.  k-Means clustering with a new divergence-based distance metric: Convergence and performance analysis , 2017, Pattern Recognit. Lett..

[5]  R. Hamming.  The Unreasonable Effectiveness of Mathematics. , 1980 .

[6]  B. Matthews.  Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.

[7]  Adam Glowacz,et al.  Fault diagnosis of single-phase induction motor based on acoustic signals , 2019, Mechanical Systems and Signal Processing.

[8]  Julio Barbancho,et al.  Non-sequential automatic classification of anuran sounds for the estimation of climate-change indicators , 2018, Expert Syst. Appl..

[9]  José Salvador Sánchez,et al.  Index of Balanced Accuracy: A Performance Measure for Skewed Class Distributions , 2009, IbPRIA.

[10]  M. Xiong,et al.  Symmetry-based structure entropy of complex networks , 2007, 0710.0108.

[11]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[12]  Peter A. Flach.  The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics , 2003, ICML.

[13]  Lin Wu,et al.  Unsupervised Metric Fusion Over Multiview Data by Graph Random Walk-Based Cross-View Diffusion , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[15]  Luc Van Gool,et al.  Computational Symmetry in Computer Vision and Computer Graphics , 2010, Found. Trends Comput. Graph. Vis..

[16]  Catherine B. Hurley,et al.  Advances in Dendrogram Seriation for Application to Visualization , 2015 .

[17]  Amir Hussain,et al.  Comparing Oversampling Techniques to Handle the Class Imbalance Problem: A Customer Churn Prediction Case Study , 2016, IEEE Access.

[18]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[19]  Julio Barbancho,et al.  Optimal Representation of Anuran Call Spectrum in Environmental Monitoring Systems Using Wireless Sensor Networks , 2018, Sensors.

[20]  Ling Shao,et al.  Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval , 2018, IEEE Transactions on Image Processing.

[21]  Lin Wu,et al.  Robust Subspace Clustering for Multi-View Data by Exploiting Correlation Consensus , 2015, IEEE Transactions on Image Processing.

[22]  J. Siegrist.  Symmetry in social exchange and health , 2005, European Review.

[23]  Rich Caruana,et al.  Data mining in metric space: an empirical analysis of supervised learning performance criteria , 2004, ROCAI.

[24]  José Hernández-Orallo,et al.  An experimental comparison of performance measures for classification , 2009, Pattern Recognit. Lett..

[25]  M. N. Sulaiman,et al.  A Review On Evaluation Metrics For Data Classification Evaluations , 2015 .

[26]  Adam Glowacz,et al.  Acoustic-Based Fault Diagnosis of Commutator Motor , 2018, Electronics.

[27]  Ya-Fen Chang,et al.  Separable Reversible Data Hiding in Encrypted Signals with Public Key Cryptography , 2018, Symmetry.

[28]  Nikolaos M. Avouris,et al.  EVALUATION OF CLASSIFIERS FOR AN UNEVEN CLASS DISTRIBUTION PROBLEM , 2006, Appl. Artif. Intell..

[29]  Pengfei Zhang,et al.  Machine Learning Topological Invariants with Neural Networks , 2017, Physical review letters.

[30]  Fernando De la Torre,et al.  Facing Imbalanced Data--Recommendations for the Use of Performance Metrics , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[31]  Angel Garrido.  Symmetry and Asymmetry Level Measures , 2010, Symmetry.

[32]  Jan Gorodkin,et al.  Comparing two K-category assignments by a K-category correlation coefficient , 2004, Comput. Biol. Chem..

[33]  Shmuel Raz,et al.  Fluctuating Asymmetry of Plant Leaves: Batch Processing with LAMINA and Continuous Symmetry Measures , 2015, Symmetry.

[34]  Cesare Furlanello,et al.  A Comparison of MCC and CEN Error Measures in Multi-Class Prediction , 2010, PloS one.

[35]  Shyr-Shen Yu,et al.  Distance Metric Based Oversampling Method for Bioinformatics and Performance Evaluation , 2016, Journal of Medical Systems.

[36]  Arezoo Islami,et al.  A match not made in heaven: on the applicability of mathematics in physics , 2016, Synthese.

[37]  Kai Ming Ting,et al.  Confusion Matrix , 2010, Encyclopedia of Machine Learning and Data Mining.

[38]  Margrit Betke,et al.  A Human–Computer Interface Using Symmetry Between Eyes to Detect Gaze Direction , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[39]  Stan Szpakowicz,et al.  Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation , 2006, Australian Conference on Artificial Intelligence.

[40]  M. Aly.  Survey on Multiclass Classification Methods , 2005 .

[41]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[42]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[43]  Anselm Brachmann,et al.  Using Convolutional Neural Network Filters to Measure Left-Right Mirror Symmetry in Images , 2016, Symmetry.