Asymptotic Analysis of an Ensemble of Randomly Projected Linear Discriminants

Datasets from the fields of bioinformatics, chemometrics, and face recognition are typically characterized by small samples of high-dimensional data. Among the many variants of linear discriminant analysis that have been proposed in order to rectify the issues associated with classification in such a setting, the classifier in (Durrant and Kabán, 2013), composed of an ensemble of randomly projected linear discriminants, seems especially promising; it is computationally efficient and, with the optimal projection dimension parameter setting, is competitive with the state-of-the-art. In this work, we seek to further understand the behavior of this classifier through asymptotic analysis. Under the assumption of a growth regime in which the dataset and projection dimensions grow at constant rates to each other, we use random matrix theory to derive asymptotic misclassification probabilities showing the effect of the ensemble as a regularization of the data sample covariance matrix. The asymptotic errors further help to identify situations in which the ensemble offers a performance advantage. We also develop a consistent estimator of the misclassification probability as an alternative to the computationally-costly cross-validation estimator, which is conventionally used for parameter tuning. Finally, we demonstrate the use of our estimator for tuning the projection dimension on both real and synthetic data.

[1]  Robert P. W. Duin,et al.  Expected classification error of the Fisher linear classifier with pseudo-inverse covariance matrix , 1998, Pattern Recognit. Lett..

[2]  Eric Jukes Encyclopedia of Machine Learning and Data Mining (2nd edition) , 2018 .

[3]  Thomas L. Marzetta,et al.  A Random Matrix-Theoretic Approach to Handling Singular Covariance Estimates , 2011, IEEE Transactions on Information Theory.

[4]  R. Tibshirani,et al.  Penalized Discriminant Analysis , 1995 .

[5]  Konstantinos N. Plataniotis,et al.  Face recognition using LDA-based algorithms , 2003, IEEE Trans. Neural Networks.

[6]  Mohamed-Slim Alouini,et al.  A Large Dimensional Study of Regularized Discriminant Analysis , 2017, IEEE Transactions on Signal Processing.

[7]  Raúl A. Gil,et al.  Chemometric application in foodomics: Nutritional quality parameters evaluation in milk-based infant formula , 2017 .

[8]  Pedro M. Domingos A few useful things to know about machine learning , 2012, Commun. ACM.

[9]  Edward R. Dougherty,et al.  Generalized Consistent Error Estimator of Linear Discriminant Analysis , 2015, IEEE Transactions on Signal Processing.

[10]  A. Robert Calderbank,et al.  Asymptotic Performance of Linear Discriminant Analysis with Random Projections , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Edward R. Dougherty,et al.  Small Sample Issues for Microarray-Based Classification , 2001, Comparative and functional genomics.

[12]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[13]  R. Samworth,et al.  Random‐projection ensemble classification , 2015, 1504.04595.

[14]  Ata Kabán,et al.  Random Projections as Regularizers: Learning a Linear Discriminant Ensemble from Fewer Observations than Dimensions , 2013, ACML.

[15]  Ata Kabán,et al.  A Bound on the Performance of LDA in Randomly Projected Data Spaces , 2010, 2010 20th International Conference on Pattern Recognition.

[16]  Mohamed-Slim Alouini,et al.  Regularized Discriminant Analysis: A Large Dimensional Study , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[17]  Héctor M. Pérez Meana,et al.  A view-invariant gait recognition algorithm based on a joint-direct linear discriminant analysis , 2017, Applied Intelligence.

[18]  Binyan Jiang,et al.  On the dimension effect of regularized linear discriminant analysis , 2017, 1710.03136.

[19]  Daniel Pizarro-Perez,et al.  Computer-Aided Classification of Gastrointestinal Lesions in Regular Colonoscopy , 2016, IEEE Transactions on Medical Imaging.

[20]  Jin-Kao Hao,et al.  A hybrid LDA and genetic algorithm for gene selection and classification of microarray data , 2010, Neurocomputing.

[21]  Kuldip K. Paliwal,et al.  Linear discriminant analysis for the small sample size problem: an overview , 2014, International Journal of Machine Learning and Cybernetics.

[22]  Qing Mai,et al.  A review of discriminant analysis in high dimensions , 2013 .

[23]  Kuldip K. Paliwal,et al.  Improved direct LDA and its application to DNA microarray gene expression data , 2010, Pattern Recognit. Lett..

[24]  Ata Kabán On Compressive Ensemble Induced Regularisation: How Close is the Finite Ensemble Precision Matrix to the Infinite Ensemble? , 2017, ALT.

[25]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[26]  Philippe Loubaton,et al.  ON BILINEAR FORMS BASED ON THE RESOLVENT OF LARGE RANDOM MATRICES , 2010, 1004.3848.

[27]  Xiaohui Wei,et al.  Fisher Discrimination Regularized Robust Coding Based on a Local Center for Tumor Classification , 2018, Scientific Reports.

[28]  A. Young,et al.  Understanding face familiarity , 2018, Cognition.

[29]  Kuldip K. Paliwal,et al.  Cancer classification by gradient LDA technique using microarray gene expression data , 2008, Data Knowl. Eng..

[30]  Robert J. Durrant,et al.  Learning in high dimensions with projected linear discriminants , 2013 .

[31]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.

[32]  M. Debbah,et al.  Random Matrix Theory Tutorial – Introduction to Deterministic Equivalents , 2015 .

[33]  L. Giansante,et al.  Classification of monovarietal Italian olive oils by unsupervised (PCA) and supervised (LDA) chemometrics , 2003 .

[34]  S. Dudoit,et al.  Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data , 2002 .

[35]  S. Girotti,et al.  Checking syrup adulteration of honey using bioluminescent bacteria and chemometrics , 2018, European Food Research and Technology.

[36]  Mohamed-Slim Alouini,et al.  A Large Dimensional Analysis of Regularized Discriminant Analysis Classifiers , 2017 .

[37]  V. Pasa,et al.  Use of Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) in Gas Chromatographic (GC) Data in the Investigation of Gasoline Adulteration , 2007 .

[38]  Satoru Miyano,et al.  A filter based feature selection algorithm using null space of covariance matrix for DNA microarray gene expression data , 2012 .

[39]  Yoshua Bengio,et al.  No Unbiased Estimator of the Variance of K-Fold Cross-Validation , 2003, J. Mach. Learn. Res..

[40]  E. Lander,et al.  Gene expression correlates of clinical prostate cancer behavior. , 2002, Cancer cell.

[41]  R. Couillet,et al.  Spectral analysis of the Gram matrix of mixture models , 2015, 1510.03463.