Elements of Information Theory

The Information Theory (IT) of Fisher and Shannon provides convenient tools for the systematic and unbiased extraction of the chemical interpretation of the known (experimental or calculated) electron distribution in a molecule. A short overview of the basic concepts, relations, and techniques of IT is presented. The Shannon (S) entropy, reflecting the amount of the uncertainty (spread, disorder) contained in the given probability distribution, and the complementary Fisher (F) (intrinsic-accuracy) measure, focusing on the distribution narrowness (order), are introduced. The relative (“cross”) entropy (entropy deficiency, missing information, directed-divergence) concept of Kullback and Leibler (KL), probing the information distance between the compared probability distributions, is presented. Rudiments of the IT descriptors of the communication channels are outlined and applied to the illustrative symmetric binary channel (SBC). The average conditional-entropy (communication noise) and mutual-information (information flow) quantities of information networks are then discussed in a more detail in view of their importance for interpreting the covalent and ionic bond components within the “communication” theory of the chemical bond. The information characteristics or several dependent probability schemes are then briefly summarized and the variational principle for the constrained extremum of the adopted measure of information, called the extreme physical information (EPI) principle, is advocated as a powerful tool for an unbiased assimilation in the optimum probability distribution of the information contained in the relevant constraints and/or references.

[1]  R. Nalewajski.  Molecular communication channels of model excited electron configurations , 2006 .

[2]  Dariusz W. Szczepanik,et al.  Bond differentiation and orbital decoupling in the orbital-communication theory of the chemical bond , 2011 .

[3]  R. Fisher,et al.  On the Mathematical Foundations of Theoretical Statistics , 1922 .

[4]  C. Weizsäcker.  Zur Theorie der Kernmassen , 1935 .

[5]  R. Nalewajski.  Entropy descriptors of the chemical bond in information theory. II. Applications to simple orbital models , 2004 .

[6]  R. Nalewajski.  Entropy/information bond indices of molecular fragments , 2005 .

[7]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[8]  Partial communication channels of molecular fragments and their entropy/information indices , 2005 .

[9]  E. Broniatowska,et al.  Entropy displacement and information distance analysis of electron distributions in molecules and their Hirshfeld atoms , 2003 .

[10]  István Mayer,et al.  Charge, bond order and valence in the AB initio SCF theory , 1983 .

[11]  R. Nalewajski.  Entropy/information coupling between orbital-communications in molecular subsystems , 2010 .

[12]  Arak M. Mathai,et al.  Basic Concepts in Information Theory and Statistics: Axiomatic Foundations and Applications , 1975 .

[13]  Sagar,et al.  Physical interpretation of information entropy: Numerical evidence of the Collins conjecture. , 1996, Physical review. A, Atomic, molecular, and optical physics.

[14]  R. Nalewajski.  Entropic bond indices from molecular information channels in orbital resolution: ground-state systems , 2008 .

[15]  R. Nalewajski.  Information principles in the loge theory , 2003 .

[16]  R. Nalewajski,et al.  Hartree‐Fock difference approach to chemical valence: Three‐electron indices in UHF approximation , 1996 .

[17]  R. Nalewajski.  Chemical bonds through probability scattering: Information channels for intermediate-orbital stages , 2008 .

[18]  R. Nalewajski.  Entropic bond indices from molecular information channels in orbital resolution: excited configurations , 2006 .

[19]  R. Parr,et al.  On the Quantum‐Mechanical Kinetic Energy as a Measure of the Information in a Distribution , 1980 .

[20]  F. L. Hirshfeld.  Bonded-atom fragments for describing molecular charge densities , 1977 .

[21]  J. Korchowiec,et al.  Reactivity criteria in charge sensitivity analysis , 1996 .

[22]  M. Gopinathan,et al.  Valency. I. A quantum chemical definition and properties , 1983 .

[23]  R. Nalewajski,et al.  Two-electron valence indices from the Kohn-Sham orbitals , 1997 .

[24]  R. Nalewajski.  Comparison between valence bond and communication theories of the chemical bond in H2 , 2006 .

[25]  R. Nalewajski.  Chemical bond descriptors from molecular information channels in orbital resolution , 2009 .

[26]  R. Parr,et al.  Information theory, atoms in molecules, and molecular similarity. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[27]  R. Nalewajski.  Information Theoretic Approach to Fluctuations and Electron Flows between Molecular Fragments , 2003 .

[28]  R. Nalewajski.  Chapter 5 Multiple, Localized, and Delocalized/Conjugated Bonds in the Orbital Communication Theory of Molecular Systems , 2009 .

[29]  R. Nalewajski.  Communication Theory Approach to the Chemical Bond , 2004 .

[30]  R. Nalewajski.  Use of non-additive information measures in exploring molecular electronic structure: stockholder bonded atoms and role of kinetic energy in the chemical bond , 2010 .

[31]  R. Nalewajski.  Reduced communication channels of molecular fragments and their entropy/information bond indices , 2005 .

[32]  H. Callen,et al.  Thermodynamics : an introduction to the physical theories of equilibrium thermostatics and irreversible thermodynamics. , 1966 .

[33]  R. Hartley.  Transmission of information , 1928 .

[34]  R. Nalewajski.  Communication-theory perspective on valence-bond theory , 2009 .

[35]  R. Nalewajski.  Orbital resolution of molecular information channels and stockholder atoms , 2006 .

[36]  R. Parr,et al.  Information Theory Thermodynamics of Molecules and Their Hirshfeld Fragments , 2001 .

[37]  Rajeev D. Bendale,et al.  Maximization of atomic information‐entropy sum in configuration and momentum spaces , 1985 .

[38]  Roman F. Nalewajski,et al.  Use of Fisher information in quantum chemistry , 2008 .

[39]  A. Köster,et al.  Electron localization function as information measure. , 2005, The journal of physical chemistry. A.

[40]  R. Nalewajski.  Hirshfeld analysis of molecular densities: subsystem probabilities and charge sensitivities , 2002 .

[41]  C. R. Smith,et al.  Maximum-Entropy and Bayesian Methods in Inverse Problems , 1985 .

[42]  E. Broniatowska,et al.  Entropy/information indices of the “stockholder” atoms‐in‐molecules , 2005 .

[43]  S. P. Gejji,et al.  Analysis of atomic electron momentum densities: Use of information entropies in coordinate and momentum space , 1985 .

[44]  R. Nalewajski.  Electronic chemical potential as information temperature , 2006 .

[45]  R. Nalewajski.  Entropic Measures of Bond Multiplicity from the Information Theory , 2000 .

[46]  R. Nalewajski,et al.  Modified valence indices from the two‐particle density matrix , 1994 .

[47]  R. Nalewajski.  Manifestations of Pauli exclusion principle in communication-theory of the chemical bond , 2009 .

[48]  G. Mazur,et al.  Quantum chemical valence indices from the one-determinantal difference approach , 1996 .

[49]  C. Bernstein,et al.  Chemical Dynamics via Molecular Beam and Laser Techniques: The Hinshelwood Lectures, Oxford, 1980 , 1982 .

[50]  Sears,et al.  Some novel characteristics of atomic information entropies. , 1985, Physical review. A, General physics.

[51]  S. Sears,et al.  An application of information theory to Compton profiles , 1979 .

[52]  Use of non-additive Fisher information in probing chemical bonds , 2010 .

[53]  A. Varandas,et al.  QUANTUM MECHANICAL VALENCE STUDY OF A BOND-BREAKING-BOND-FORMING PROCESS IN TRIATOMIC SYSTEMS , 1994 .

[54]  Roman F. Nalewajski,et al.  Entropy descriptors of the chemical bond in information theory. I. Basic concepts and relations , 2004 .

[55]  Roman F. Nalewajski,et al.  Information Theory of Molecular Systems , 2006 .

[56]  R. Nalewajski,et al.  Information distance analysis of molecular electron densities , 2002 .

[57]  P. Dirac.  Principles of Quantum Mechanics , 1982 .

[58]  B. Frieden,et al.  Weighted Fisher informations, their derivation and use , 2010 .

[59]  Solomon Kullback,et al.  Information Theory and Statistics , 1960 .

[60]  R. Nalewajski.  Atomic resolution of bond descriptors in the two-orbital model , 2006 .

[61]  Shridhar R. Gadre,et al.  Information entropy and Thomas-Fermi theory , 1984 .

[62]  Aleksandr Yakovlevich Khinchin,et al.  Mathematical foundations of information theory , 1959 .

[63]  Rory A. Fisher,et al.  Theory of Statistical Estimation , 1925, Mathematical Proceedings of the Cambridge Philosophical Society.

[64]  R. Nalewajski.  Information origins of the chemical bond: Bond descriptors from molecular communication channels in orbital resolution , 2009 .

[65]  R. Nalewajski.  Additive and non-additive information channels in orbital communication theory of the chemical bond , 2010 .

[66]  R. Nalewajski.  Entropic descriptors of the chemical bond in H2: local resolution of stockholder atoms , 2009 .

[67]  L. Brillouin,et al.  Science and information theory , 1956 .

[68]  Á. Nagy,et al.  Remarks on density functional theory as a thermodynamics , 2000 .

[69]  Norman Abramson,et al.  Information theory and coding , 1963 .

[70]  E. Broniatowska,et al.  Atoms-in-molecules from the Stockholder Partition of the Molecular Two-electron Distribution , 2006 .

[71]  INFORMATION ENTROPY AS A MEASURE OF THE QUALITY OF AN APPROXIMATE ELECTRONIC WAVE FUNCTION , 1998 .

[72]  A. Savin,et al.  Classification of chemical bonds based on topological analysis of electron localization functions , 1994, Nature.

[73]  I. Mayer.  Bond orders and valences in the SCF theory: a comment , 1985 .

[74]  Andreas Savin,et al.  ELF: The Electron Localization Function , 1997 .

[75]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[76]  A. Köster,et al.  Chemical valence from the two-particle density matrix , 1993 .

[77]  Claude E. Shannon,et al.  A mathematical theory of communication , 1948, MOCO.

[78]  R. Nalewajski.  Information-scattering perspective on orbital hybridization. , 2007, The journal of physical chemistry. A.

[79]  L. E. Clarke,et al.  Concepts of Probability Theory , 1980 .

[80]  R. Nalewajski.  Many-orbital probabilities and their entropy/information descriptors in orbital communication theory of the chemical bond , 2010 .

[81]  Axel D. Becke,et al.  A Simple Measure of Electron Localization in Atomic and Molecular-Systems , 1990 .

[82]  R. Nalewajski.  Probing the Interplay between Electronic and Geometric Degrees-of-Freedom in Molecules and Reactive Systems , 2006 .

[83]  Paul W Ayers,et al.  What is an atom in a molecule? , 2005, The journal of physical chemistry. A.

[84]  R. Nalewajski.  Applications of the Information Theory to Problems of Molecular Electronic Structure and Chemical Reactivity , 2002 .

[85]  Kenneth B. Wiberg,et al.  Application of the pople-santry-segal CNDO method to the cyclopropylcarbinyl and cyclobutyl cation and to bicyclobutane , 1968 .

[86]  R. Nalewajski.  Entropic and difference bond multiplicities from the two-electron probabilities in orbital resolution , 2004 .