Information-Theoretic Learning Using Renyi's Quadratic Entropy

Learning from examples has been traditionally based on correlation or on the mean square error (MSE) criterion, in spite of the fact that learning is intrinsically related with the extraction of information from examples. The problem is that Shannon’s Information Entropy, which has a sound theoretical foundation, is not easy to implement in a learning from examples scenario. In this paper, Renyi’s Entropy definition is used and integrated with a nonparametric estimator of the probability density function (Parzen Window). We develop a physical analogy for the estimation method which we call the information potential field. The “information potential” method is rather general and will have many applications.

[1]  Mill Johannes G.A. Van,et al.  Transmission Of Information , 1961 .

[2]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[3]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[4]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[5]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[6]  Jagat Narain Kapur,et al.  Measures of information and their applications , 1994 .

[7]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[8]  Paul A. Viola,et al.  Empirical Entropy Manipulation for Real-World Problems , 1995, NIPS.

[9]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[10]  G. Deco,et al.  An Information-Theoretic Approach to Neural Computing , 1997, Perspectives in Neural Computing.

[11]  J. Príncipe,et al.  Entropy manipulation of arbitrary nonlinear mappings , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[12]  Vladimir Cherkassky,et al.  The Nature Of Statistical Learning Theory , 1997, IEEE Trans. Neural Networks.

[13]  John W. Fisher,et al.  Pose estimation in SAR using an information theoretic criterion , 1998, Defense, Security, and Sensing.

[14]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.