On Bayesian Inference, Maximum Entropy and Support Vector Machines Methods

The analysis of discrimination, feature and model selection conduct to the discussion of the relationships between Support Vector Machine (SVM), Bayesian and Maximum Entropy (MaxEnt) formalisms. MaxEnt discrimination can be seen as a particular case of Bayesian inference, which at its turn can be seen as a regularization approach applicable to SVM. Probability measures can be attached to each feature vector, thus feature selection can be described by a discriminative model over the feature space. Further the probabilistic SVM allows to define a posterior probability model for a classifier. In addition, the similarities with the kernels based on Kullback‐Leibler divergence can be deduced, thus returning with MaxEnt similarity.

[1]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[2]  S. Gull,et al.  Maximum entropy algorithm applied to image enhancement , 1980 .

[3]  Christopher M. Bishop,et al.  Bayesian Regression and Classification , 2003 .

[4]  Arun D Kulkarni,et al.  Neural Networks for Pattern Recognition , 1991 .

[5]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[6]  S. Sathiya Keerthi,et al.  A new Bayesian design method for support vector classification , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[7]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[8]  E. Jaynes.  On the rationale of maximum-entropy methods , 1982, Proceedings of the IEEE.

[9]  Ingemar J. Cox,et al.  The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments , 2000, IEEE Trans. Image Process..

[10]  Bernhard Schölkopf,et al.  Comparing support vector machines with Gaussian kernels to radial basis function classifiers , 1997, IEEE Trans. Signal Process..

[11]  T. J. Ponman,et al.  Maximum entropy methods , 1984 .

[12]  David J. C. MacKay,et al.  The Evidence Framework Applied to Classification Networks , 1992, Neural Computation.

[13]  Nuno Vasconcelos,et al.  A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications , 2003, NIPS.

[14]  John Skilling,et al.  Maximum entropy method in image processing , 1984 .

[15]  Mihai Datcu,et al.  Categorization based Relevance Feedback Search Engine for Earth Observation Images Repositories , 2006, 2006 IEEE International Symposium on Geoscience and Remote Sensing.

[16]  Matthew L. Miller,et al.  The Bayesian Image Retrieval System, PicHunter , 2000 .

[17]  John Skilling,et al.  Image restoration by a powerful maximum entropy method , 1982, Comput. Graph. Image Process..

[18]  Marin Ferecatu,et al.  Tuning SVM-Based Relevance Feedback for the Interactive Classification of Images , 2004, EWIMT.