Kolmogorov Complexity and Information Theory. With an Interpretation in Terms of Questions and Answers

We compare the elementary theories of Shannon information and Kolmogorov complexity, the extent to which they have a common purpose, and wherethey are fundamentally different. We discuss and relate the basicnotions of both theories: Shannon entropy, Kolmogorov complexity, Shannon mutual informationand Kolmogorov (``algorithmic'') mutual information. We explainhow universal coding may be viewed as a middle ground betweenthe two theories. We consider Shannon's rate distortion theory, whichquantifies useful (in a certain sense) information.We use the communication of information as our guiding motif, and we explain howit relates to sequential question-answer sessions.

[1]  Ming Li,et al.  Minimum description length induction, Bayesianism, and Kolmogorov complexity , 1999, IEEE Trans. Inf. Theory.

[2]  R. Fisher,et al.  On the Mathematical Foundations of Theoretical Statistics , 1922 .

[3]  Gregory J. Chaitin,et al.  Algorithmic Information Theory , 1987, IBM J. Res. Dev..

[4]  Nikolai K. Vereshchagin,et al.  Kolmogorov's structure functions with an application to the foundations of model selection , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[5]  Neri Merhav,et al.  Universal Prediction , 1998, IEEE Trans. Inf. Theory.

[6]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[7]  Robert van Rooy,et al.  Quality and Quantity of Information Exchange , 2003, J. Log. Lang. Inf..

[8]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[9]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[10]  A. Kolmogorov.  Three approaches to the quantitative definition of information , 1968 .

[11]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[12]  Claude E. Shannon,et al.  The Mathematical Theory of Communication , 1950 .

[13]  Jorma Rissanen.  Kolmogorov's structure function for probability models , 2002, Proceedings of the IEEE Information Theory Workshop.

[14]  Péter Gács,et al.  Algorithmic statistics , 2000, IEEE Trans. Inf. Theory.