Autoencoders trained with relevant information: Blending Shannon and Wiener's perspectives

It is almost seventy years after the publication of Claude Shannon's “A Mathematical Theory of Communication” [1] and Norbert Wiener's “Extrapolation, Interpolation and Smoothing of Stationary Time Series” [2]. The pioneering works of Shannon and Wiener lay the foundation of communication, data storage, control, and other information technologies. This paper briefly reviews Shannon and Wiener's perspectives on the problem of message transmission over noisy channel and also experimentally evaluates the feasibility of integrating these two perspectives to train autoencoders close to the information limit. To this end, the principle of relevant information (PRI) is used and validated to optimally encode input imagery in the presence of noise.

[1]  J. Príncipe,et al.  Nonlinear extensions to the minimum average correlation energy filter , 1997 .

[2]  S LuisGonzalo.  REPRODUCING KERNEL HILBERT SPACE METHODS FOR INFORMATION THEORETIC LEARNING , 2012 .

[3]  S. Kak.  Information, physics, and computation , 1996 .

[4]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[5]  Sudhir Madhav Rao,et al.  Unsupervised learning: An information theoretic framework , 2008 .

[6]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series , 1964 .

[7]  Yizong Cheng,et al.  Mean Shift, Mode Seeking, and Clustering , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[9]  Eder Santana,et al.  Information Theoretic-Learning auto-encoder , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[10]  King-Sun Fu,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Publication Information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[12]  Thomas Kailath,et al.  A view of three decades of linear filtering theory , 1974, IEEE Trans. Inf. Theory.

[13]  Pierre Baldi,et al.  Autoencoders, Unsupervised Learning, and Deep Architectures , 2011, ICML Unsupervised and Transfer Learning.

[14]  Jose C. Principe,et al.  Information Theoretic Learning - Renyi's Entropy and Kernel Perspectives , 2010, Information Theoretic Learning.

[15]  Sergio Verdú,et al.  Fifty Years of Shannon Theory , 1998, IEEE Trans. Inf. Theory.

[16]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[17]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[18]  R. Blahut.  Algebraic Codes for Data Transmission , 2002 .

[19]  J.E. Mazo,et al.  Digital communications , 1985, Proceedings of the IEEE.

[20]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.