An introduction to information theory and entropy

information theory and entropy Tom Carter http:// ogs. sustan.edu/~ tom/information Complex Systems Summer S hool June, 2000 1 Our general topi s: } Measuring omplexity } Some probability ba kground } Basi s of information theory } Some entropy theory } The Gibbs inequality } A simple physi al example (gases) } Shannon's ommuni ation theory } Analog hannels } Appli ation to Biology (analyzing genomes) } Appli ation to Physi s (lasers) } Some other measures } Future dire tions and prospe ts } Referen es 2 The quotes } S ien e, wisdom, and ounting } Being di erent { or random } Surprise, information, and mira les } Information (and hope) } H (or S) for Entropy } Thermodynami s } Language, and putting things together } Tools

[1]  J.,et al.  Using DNA to Solve NP-Complete ProblemsRichard , 1995 .

[2]  H. Rechenberg,et al.  Niels Bohr (1885–1962) , 1985 .

[3]  Rolf Landauer,et al.  Information is Physical , 1991, Workshop on Physics and Computation.

[4]  Jacobus H. van Lint,et al.  Coding Theory , 1971 .

[5]  W. H. Zurek,et al.  Thermodynamic cost of computation, algorithmic complexity and the information metric , 1989, Nature.

[6]  Peter H. Richter.  Information and Self-organization: A Macroscopic Approach to Complex Systems, Hermann Haken. Springer, New York (1988), $59.50 (cloth), 196 pp , 1991 .

[7]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[8]  Edmund Whittaker.  Albert Einstein, 1879-1955 , 1955, Biographical Memoirs of Fellows of the Royal Society.

[9]  N. Martin,et al.  Mathematical Theory of Entropy , 1981 .

[10]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[11]  Ray Hill,et al.  A First Course in Coding Theory , 1988 .

[12]  Anthony J. G. Hey,et al.  Feynman Lectures on Computation , 1996 .

[13]  D. S. Jones,et al.  Elementary information theory , 1979 .

[14]  David Slepian,et al.  Key papers in the development of information theory , 1974 .

[15]  Andrew Hodges,et al.  Alan Turing: The Enigma , 1983 .

[16]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[17]  J. Hailman.  Evolution as Entropy: Toward a Unified Theory of Biology, Daniel R. Brooks, E.O. Wiley. University of Chicago Press, Chicago (1986), xiv, +335. Price $25.00 , 1987 .

[18]  J. von Neumann,et al.  Probabilistic Logic and the Synthesis of Reliable Organisms from Unreliable Components , 1956 .

[19]  D. Hofstadter.  Metamagical Themas: Questing for the Essence of Mind and Pattern , 1985 .

[20]  M. V. van Emden.  Adaptive information processing: An introductory survey , 1978, Proceedings of the IEEE.

[21]  Manfred Schroeder,et al.  Fractals, Chaos, Power Laws: Minutes From an Infinite Paradise , 1992 .

[22]  Daniel R. Brooks,et al.  Evolution as entropy: Toward a unified theory of biology , 1987 .

[23]  Lila L. Gatlin,et al.  Information theory and the living system , 1972 .

[24]  Yaneer Bar-Yam,et al.  Dynamics Of Complex Systems , 2019 .

[25]  L. Brillouin,et al.  Science and information theory , 1956 .

[26]  L. Turner.  Key Papers in the Development of Information Theory , 1975 .

[27]  A. Turing.  On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[28]  Richard Wesley Hamming,et al.  Coding and information theory (2. ed.) , 1986 .

[29]  Wolfgang Christian,et al.  Dynamics of Complex Systems (Studies in Nonlinearity) , 1998 .

[30]  O. Antoine,et al.  Theory of Error-correcting Codes , 2022 .

[31]  Steven Roman.  Introduction to coding and information theory , 1997, Undergraduate texts in mathematics.

[32]  M. H. van Emden.  Adaptive information processing: An introductory survey , 1978 .

[33]  Richard W. Hamming,et al.  Coding and Information Theory , 2018, Feynman Lectures on Computation.

[34]  R. Landauer.  The physical nature of information , 1996 .

[35]  Richard W. Hamming,et al.  Error detecting and error correcting codes , 1950 .

[36]  Jeremy Campbell,et al.  Grammatical Man: Information, Entropy, Language and Life , 1982 .

[37]  Fazlollah M. Reza,et al.  Introduction to Information Theory , 2004, Lecture Notes in Electrical Engineering.