Linear vs. quadratic discriminant analysis classifier: a tutorial

The aim of this paper is to collect in one place the basic background needed to understand the discriminant analysis (DA) classifier to make the reader of all levels be able to get a better understanding of the DA and to know how to apply this classifier in different applications. This paper starts with basic mathematical definitions of the DA steps with visual explanations of these steps. Moreover, in a step-by-step approach, a number of numerical examples were illustrated to show how to calculate the discriminant functions and decision boundaries when the covariance matrices of all classes were common or not. The singularity problem of DA was explained and some of the state-of-the-art solutions to this problem were highlighted with numerical illustrations. An experiment is conducted to compare between the linear and quadratic classifiers and to show how to solve the singularity problem when high-dimensional datasets are used.

[1]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[2]  Jammalamadaka.  Introduction to Linear Regression Analysis (3rd ed.) , 2003 .

[3]  George A. F. Seber,et al.  Linear regression analysis , 1977 .

[4]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[5]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[6]  Robin Thompson,et al.  Graphical models in applied multivariate statistics , 1992 .

[7]  Konstantinos N. Plataniotis,et al.  Regularization studies of linear discriminant analysis in small sample size scenarios with application to face recognition , 2005, Pattern Recognit. Lett..

[8]  Edward I. Altman,et al.  Corporate distress diagnosis: Comparisons using linear discriminant analysis and neural networks (the Italian experience) , 1994 .

[9]  Benjamin Naumann,et al.  Learning And Soft Computing Support Vector Machines Neural Networks And Fuzzy Logic Models , 2016 .

[10]  Hui Gao,et al.  Why direct LDA is not equivalent to LDA , 2006, Pattern Recognit..

[11]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[12]  Gerald Schaefer,et al.  Ear Recognition Using Block-Based Principal Component Analysis and Decision Fusion , 2015, PReMI.

[13]  A. Tharwat,et al.  Personal identification using ear images based on fast and accurate principal component analysis , 2012, 2012 8th International Conference on Informatics and Systems (INFOS).

[14]  M. A. Carreira-Perpinan,et al.  Compression neural networks for feature extraction: Application to human recognition from ear images , 1995 .

[15]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[16]  Trevor Hastie,et al.  Regularized linear discriminant analysis and its application in microarrays. , 2007, Biostatistics.

[17]  Alejandro F. Frangi,et al.  Two-dimensional PCA: a new approach to appearance-based face representation and recognition , 2004 .

[18]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[19]  Hua Yu,et al.  A direct LDA algorithm for high-dimensional data - with application to face recognition , 2001, Pattern Recognit..

[20]  D. Farina,et al.  Linear and Nonlinear Regression Techniques for Simultaneous and Proportional Myoelectric Control , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[21]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[22]  Kuldip K. Paliwal,et al.  Linear discriminant analysis for the small sample size problem: an overview , 2014, International Journal of Machine Learning and Cybernetics.

[23]  Alaa Tharwat.  Principal component analysis - a tutorial , 2016, Int. J. Appl. Pattern Recognit..

[24]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[25]  A. Christopoulos,et al.  Fitting Models to Biological Data Using Linear and Nonlinear Regression: A Practical Guide to Curve Fitting , 2004 .

[26]  Jieping Ye,et al.  Computational and Theoretical Analysis of Null Space and Orthogonal Linear Discriminant Analysis , 2006, J. Mach. Learn. Res..

[27]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[28]  Theofanis Sapatinas,et al.  Discriminant Analysis and Statistical Pattern Recognition , 2005 .