Some Observations on the Concepts of Information-Theoretic Entropy and Randomness

Jonathan D.H. SmithDepartment of MathematicsIowa State UniversityAmes, IA 50011, USAE-mail: jdhsmith@math.iastate.eduURL: http://www.math.iastate.edu/jdhsmith/Received: 15 February 2000 / Accepted: 11 January 2001 / Published: 1 February 2001Abstract: Certain aspects of the history, derivation, and physical application of theinformation-theoretic entropy concept are discussed. Pre-dating Shannon, the conceptis traced back to Pauli. A derivation from rst principles is given, without use ofapproximations. The concept depends on the underlying degree of randomness. Inphysical applications, this translates to dependence on the experimental apparatusavailable. An example illustrates how this dependence a ects Prigogine’s proposal forthe use of the Second Law of Thermodynamics as a selection principle for the breakingof time symmetry. The dependence also serves to yield a resolution of the so-called\Gibbs Paradox." Extension of the concept from the discrete to the continuous caseis discussed. The usual extension is shown to be dimensionally incorrect. Correctionintroduces a reference density, leading to the concept of Kullback entropy. Practicalrelativistic considerations suggest a possible proper reference density.Keywords: information-theoretic entropy; Shannon entropy; Martin-Loef random-ness; self-delimiting algorithmic complexity; thermodynamic entropy; Second Law ofThermodynamics; selection principle; wave equation; Gibbs Paradox; dimensional anal-ysis; Kullback entropy; cross-entropy; reference density; improper prior.c 2001 by the author. Reproduction for noncommercial purposes permitted.