Discriminative learning can succeed where generative learning fails

Generative algorithms for learning classifiers use training data to separately estimate a probability model for each class. New items are then classified by comparing their probabilities under these models. In contrast, discriminative learning algorithms try to find classifiers that perform well on all the training data. We show that there is a learning problem that can be solved by a discriminative learning algorithm, but not by any generative learning algorithm (given minimal cryptographic assumptions). This statement is formalized using a framework inspired by previous work of Goldberg [3].

[1]  David Haussler,et al.  Exploiting Generative Models in Discriminative Classifiers , 1998, NIPS.

[2]  Vladimir Vapnik.  Estimations of dependences based on statistical data , 1982 .

[3]  Paul W. Goldberg.  When Can Two Unsupervised Learners Achieve PAC Separation? , 2001, COLT/EuroCOLT.

[4]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[5]  Michael I. Jordan,et al.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes , 2001, NIPS.

[6]  Tony Jebara,et al.  Machine learning: Discriminative and generative , 2006 .

[7]  V. Vapnik.  Estimation of Dependences Based on Empirical Data , 2006 .

[8]  Paul W. Goldberg,et al.  Some Discriminant-Based PAC Algorithms , 2006, J. Mach. Learn. Res..

[9]  Silvio Micali,et al.  How to construct random functions , 1986, JACM.

[10]  Oded Goldreich.  Foundations of Cryptography: Volume 1 , 2006 .

[11]  Leslie G. Valiant,et al.  A general lower bound on the number of examples needed for learning , 1988, COLT '88.

[12]  Rocco A. Servedio,et al.  Discriminative learning can succeed where generative learning fails , 2007, Inf. Process. Lett..

[13]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[14]  David G. Stork,et al.  Pattern Classification , 1973 .

[16]  Leonid A. Levin,et al.  A Pseudorandom Generator from any One-way Function , 1999, SIAM J. Comput..

[17]  Rajat Raina,et al.  Classification with Hybrid Generative/Discriminative Models , 2003, NIPS.

[18]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.