Quantifying the reliability of fault classifiers

Fault diagnostics problems can be formulated as classification tasks. Due to limited data and to uncertainty, classification algorithms are not perfectly accurate in practical applications. Maintenance decisions based on erroneous fault classifications result in inefficient resource allocations and/or operational disturbances. Thus, knowing the accuracy of classifiers is important to give confidence in the maintenance decisions. The average accuracy of a classifier on a test set of data patterns is often used as a measure of confidence in the performance of a specific classifier. However, the performance of a classifier can vary in different regions of the input data space. Several techniques have been proposed to quantify the reliability of a classifier at the level of individual classifications. Many of the proposed techniques are only applicable to specific classifiers, such as ensemble techniques and support vector machines. In this paper, we propose a meta approach based on the typicalness framework (Kolmogorov's concept of randomness), which is independent of the applied classifier. We apply the approach to a case of fault diagnosis in railway turnout systems and compare the results obtained with both extreme learning machines and echo state networks.

[1]  E. Zio,et al.  A METHOD FOR ESTIMATING THE CONFIDENCE IN THE IDENTIFICATION OF NUCLEAR TRANSIENTS BY A BAGGED ENSEMBLE OF FCM CLASSIFIERS , 2010 .

[2]  Nasser L. Azad,et al.  Optimally pruned extreme learning machine with ensemble of regularization techniques and negative correlation penalty applied to automotive engine coldstart hydrocarbon emission identification , 2014, Neurocomputing.

[3]  Daniel Barbará,et al.  Detecting outliers using transduction and statistical testing , 2006, KDD '06.

[4]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[5]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[6]  Igor Kononenko,et al.  An overview of advances in reliability estimation of individual predictions in machine learning , 2009, Intell. Data Anal..

[7]  Rob Law,et al.  Complex system fault diagnosis based on a fuzzy robust wavelet support vector classifier and an adaptive Gaussian particle swarm optimization , 2010, Inf. Sci..

[8]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[9]  Mohak Shah,et al.  Evaluating Learning Algorithms: A Classification Perspective , 2011 .

[10]  Alexander Gammerman,et al.  Prediction algorithms and confidence measures based on algorithmic randomness theory , 2002, Theor. Comput. Sci..

[11]  Khashayar Khorasani,et al.  Fault detection and isolation of a dual spool gas turbine engine using dynamic neural networks and multiple model approach , 2014, Inf. Sci..

[12]  Binu P. Chacko,et al.  Handwritten character recognition using wavelet energy and extreme learning machine , 2012, Int. J. Mach. Learn. Cybern..

[13]  Stephen M. Omohundro,et al.  Five Balltree Construction Algorithms , 2009 .

[14]  Yoshua Bengio,et al.  Gradient-Based Optimization of Hyperparameters , 2000, Neural Computation.

[15]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[16]  Benjamin Schrauwen,et al.  Recurrent Kernel Machines: Computing with Infinite Echo State Networks , 2012, Neural Computation.

[17]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.

[18]  Ethem Alpaydin,et al.  Introduction to machine learning , 2004, Adaptive computation and machine learning.

[19]  Magdi S. Mahmoud,et al.  Expectation maximization approach to data-based fault diagnostics , 2013, Inf. Sci..

[20]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[21]  Daming Lin,et al.  A review on machinery diagnostics and prognostics implementing condition-based maintenance , 2006 .

[22]  Ida G. Sprinkhuizen-Kuyper,et al.  Meta-Typicalness Approach to Reliable Classification , 2006, ECAI.

[23]  Yuan Lan,et al.  Ensemble of online sequential extreme learning machine , 2009, Neurocomputing.

[24]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[25]  Tom Heskes,et al.  Practical Confidence and Prediction Intervals , 1996, NIPS.

[26]  I. Bratko,et al.  Information-based evaluation criterion for classifier's performance , 2004, Machine Learning.

[27]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Texts in Computer Science.

[28]  Herbert Jaeger,et al.  Echo State Property Linked to an Input: Exploring a Fundamental Characteristic of Recurrent Neural Networks , 2013, Neural Computation.

[29]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[30]  Frank L. Lewis,et al.  Intelligent Fault Diagnosis and Prognosis for Engineering Systems , 2006 .

[31]  Per Martin-Löf,et al.  The Definition of Random Sequences , 1966, Inf. Control..

[32]  William I. Gasarch,et al.  Book Review: An introduction to Kolmogorov Complexity and its Applications Second Edition, 1997 by Ming Li and Paul Vitanyi (Springer (Graduate Text Series)) , 1997, SIGACT News.

[33]  Mantas Lukosevicius,et al.  A Practical Guide to Applying Echo State Networks , 2012, Neural Networks: Tricks of the Trade.

[34]  Nader Meskin,et al.  Multiple sensor fault diagnosis by evolving data-driven approach , 2014, Inf. Sci..