Joint discriminative-generative modelling based on statistical tests for classification

In statistical pattern classification, generative approaches, such as linear discriminant analysis (LDA), assume a data-generating process (DGP), whereas discriminative approaches, such as linear logistic regression (LLR), do not model the DGP. In general, a generative classifier performs better than its discriminative counterpart if the DGP is well-specified and worse than the latter if the DGP is clearly mis-specified. In view of this, this paper presents a joint discriminative-generative modelling (JoDiG) approach, by partitioning predictor variables X into two sub-vectors, namely X"G, to which a generative approach is applied, and X"D, to be treated by a discriminative approach. This partitioning of X is based on statistical tests of the assumed DGP: the variables that clearly fail the tests are grouped as X"D and the rest as X"G. Then the generative and discriminative approaches are combined in a probabilistic rather than a heuristic way. The principle of the JoDiG approach is quite generic, but for illustrative purposes numerical studies of the paper focus on a widely-used case, in which the DGP assumes a multivariate normal distribution for each class. In this case, the JoDiG approach uses LDA for X"G and LLR for X"D. Numerical experiments on real and simulated data demonstrate that the performance of this new approach to classification is similar to or better than that of its discriminative and generative counterparts, in particular when the size of the training-set is comparable to the dimension of the data.

[1]  Christina Gloeckner,et al.  Modern Applied Statistics With S , 2003 .

[2]  R. Tibshirani,et al.  Discriminant Analysis by Gaussian Mixtures , 1996 .

[3]  Robert Tibshirani,et al.  Classification by Pairwise Coupling , 1997, NIPS.

[4]  Wojtek J. Krzanowski,et al.  Stepwise Location Model Choice in Mixed‐Variable Discrimination , 1983 .

[5]  Jin Tian,et al.  A Hybrid Generative/Discriminative Bayesian Classifier , 2006, FLAIRS Conference.

[6]  B. Efron.  The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis , 1975 .

[7]  Stephen Warwick Looney,et al.  How to Use Tests for Univariate Normality to Assess Multivariate Normality , 1995 .

[8]  Guillaume Bouchard,et al.  The Tradeoff Between Generative and Discriminative Classifiers , 2004 .

[9]  Rajat Raina,et al.  Classification with Hybrid Generative/Discriminative Models , 2003, NIPS.

[10]  D. M. Titterington,et al.  On the generative-discriminative tradeoff approach: Interpretation, asymptotic efficiency and classification performance , 2010, Comput. Stat. Data Anal..

[11]  D. Titterington,et al.  Comparison of Discrimination Techniques Applied to a Complex Data Set of Head Injured Patients , 1981 .

[12]  Terence J. O'Neill.  The General Distribution of the Error Rate of a Classification Procedure With Application to Logistic Regression Discrimination , 1980 .

[13]  D. M. Titterington,et al.  Interpretation of hybrid generative/discriminative algorithms , 2009, Neurocomputing.

[14]  Michael I. Jordan,et al.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes , 2001, NIPS.

[15]  A P Dawid,et al.  Properties of diagnostic data distributions. , 1976, Biometrics.

[16]  Christopher Joseph Pal,et al.  Multi-Conditional Learning: Generative/Discriminative Training for Clustering and Classification , 2006, AAAI.

[17]  A. P. Dawid,et al.  Generative or Discriminative? Getting the Best of Both Worlds , 2007 .

[18]  Trevor J. Hastie,et al.  Discriminative vs Informative Learning , 1997, KDD.

[19]  Ryan M. Rifkin,et al.  In Defense of One-Vs-All Classification , 2004, J. Mach. Learn. Res..

[20]  D. M. Titterington,et al.  Comment on “On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes” , 2008, Neural Processing Letters.

[21]  T. Minka.  A comparison of numerical optimizers for logistic regression , 2004 .

[22]  D. Hand,et al.  Idiot's Bayes—Not So Stupid After All? , 2001 .