Emerging themes on information theory and Bayesian approach

Though efforts on the quantification of information started several decades earlier, the foundations of information theoretic studies were laid during the middle and late 1940’s, from two perspectives that both based on probability theory. The most famous one is a systematic theory from a perspective of information transmission over a noisy channel, namely the information theory developed by Claude E. Shannon [1]. The other consists of some fundamental

[1]  Lei Xu,et al.  Temporal BYY learning for state space approach, hidden Markov model, and blind source separation , 2000, IEEE Trans. Signal Process..

[2]  A. Kolmogorov.  Three approaches to the quantitative definition of information , 1968 .

[3]  H. Akaike.  A new look at the statistical model identification , 1974 .

[4]  R. Johnson,et al.  Properties of cross-entropy minimization , 1981, IEEE Trans. Inf. Theory.

[5]  D. M. Titterington,et al.  Variational approximations in Bayesian model selection for finite mixture distributions , 2007, Comput. Stat. Data Anal..

[6]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part I , 1964, Inf. Control..

[7]  N. Čencov.  Statistical Decision Rules and Optimal Inference , 2000 .

[8]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[9]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[10]  C. R. Rao,et al.  Information and the Accuracy Attainable in the Estimation of Statistical Parameters , 1992 .

[11]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[12]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[13]  A. Yuille,et al.  Opinion TRENDS in Cognitive Sciences Vol.10 No.7 July 2006 Special Issue: Probabilistic models of cognition Vision as Bayesian inference: analysis by synthesis? , 2022 .

[14]  Shun-ichi Amari,et al.  Differential-geometrical methods in statistics , 1985 .

[15]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[16]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[17]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[18]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[19]  N. N. Chent︠s︡ov.  Statistical decision rules and optimal inference , 1982 .

[20]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[21]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[22]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..