Information Theoretic Proofs of Entropy Power Inequalities

While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai, and Verdú used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.

[1]  J. Linnik.  An Information-Theoretic Proof of the Central Limit Theorem with Lindeberg Conditions , 1959 .

[2]  Shunsuke Ihara,et al.  On the Capacity of Channels with Additive Non-Gaussian Noise , 1978, Inf. Control..

[3]  E. Kuruoglu,et al.  A Minimax Entropy Method for Blind Separation of Dependent Components in Astrophysical Images , 2006 .

[4]  Andrei V. Kelarev,et al.  The Theory of Information and Coding , 2005 .

[5]  Michel Verleysen,et al.  Mixing and Non-Mixing Local Minima of the Entropy Contrast for Blind Source Separation , 2006, IEEE Transactions on Information Theory.

[6]  Suhas N. Diggavi,et al.  The worst additive noise under a covariance constraint , 2001, IEEE Trans. Inf. Theory.

[7]  Daniel Pérez Palomar,et al.  Gradient of mutual information in linear vector Gaussian channels , 2005, ISIT.

[8]  Abram Kagan,et al.  A discrete version of the Stam inequality and a characterization of the Poisson distribution , 2001 .

[9]  K. Ball,et al.  Solution of Shannon's problem on the monotonicity of entropy , 2004 .

[10]  Meir Feder,et al.  On the Volume of the Minkowski Sum of Line Sets and the Entropy-Power Inequality , 1998, IEEE Trans. Inf. Theory.

[11]  Jean-François Bercher,et al.  Estimating the entropy of a signal with applications , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[12]  A. Barron,et al.  Fisher information inequalities and the central limit theorem , 2001, math/0111020.

[13]  Yoshiaki Itoh.  The Information Theoretic Proof of Kac's Theorem , 1970 .

[14]  John M. Cioffi,et al.  A Proof of the Converse for the Capacity of Gaussian MIMO Broadcast Channels , 2006, 2006 IEEE International Symposium on Information Theory.

[15]  E.C. van der Meulen,et al.  Higher order asymptotics of mutual information for nonlinear channels with nongaussian noise , 2003, IEEE International Symposium on Information Theory, 2003. Proceedings..

[16]  Meir Feder,et al.  A Generalization of the Entropy Power Inequality with Applications to Linear Transformation of a White-Noise , 1993, Proceedings. IEEE International Symposium on Information Theory.

[17]  E. Carlen,et al.  Entropy production by block variable summation and central limit theorems , 1991 .

[18]  V. V. Prelov.  Communication Channel Capacity with Almost Gaussian Noise , 1989 .

[19]  Evgueni A. Haroutunian,et al.  Information Theory and Statistics , 2011, International Encyclopedia of Statistical Science.

[20]  E. C. van der Meulen,et al.  INFORMATION RATES IN CERTAIN STATIONARY NON-GAUSSIAN CHANNELS IN WEAK-SIGNAL TRANSMISSION , 1998 .

[21]  Sergio Verdú,et al.  On channel capacity per unit cost , 1990, IEEE Trans. Inf. Theory.

[22]  Ender Tekin,et al.  The Gaussian Multiple Access Wire-Tap Channel with Collective Secrecy Constraints , 2006, 2006 IEEE International Symposium on Information Theory.

[23]  Martin E. Hellman,et al.  The Gaussian wire-tap channel , 1978, IEEE Trans. Inf. Theory.

[24]  M. Feder,et al.  A matrix form of the Brunn-Minkowski inequality , 1995, Proceedings of 1995 IEEE International Symposium on Information Theory.

[25]  Daniel Pérez Palomar,et al.  Gradient of mutual information in linear vector Gaussian channels , 2006, IEEE Transactions on Information Theory.

[26]  Shlomo Shamai,et al.  Fading channels: How perfect need "Perfect side information" be? , 2002, IEEE Trans. Inf. Theory.

[27]  Sergio Verdú,et al.  A simple proof of the entropy-power inequality , 2006, IEEE Transactions on Information Theory.

[28]  Cédric Villani,et al.  A short proof of the "Concavity of entropy power" , 2000, IEEE Trans. Inf. Theory.

[29]  L. Ozarow,et al.  On a source-coding problem with two channels and three receivers , 1980, The Bell System Technical Journal.

[30]  A. Barron.  ENTROPY AND THE CENTRAL LIMIT THEOREM , 1986 .

[31]  Meir Feder,et al.  Rate-distortion performance in coding bandlimited sources by sampling and dithered quantization , 1995, IEEE Trans. Inf. Theory.

[32]  Shlomo Shamai,et al.  The Capacity Region of the Gaussian Multiple-Input Multiple-Output Broadcast Channel , 2006, IEEE Transactions on Information Theory.

[33]  Dinh-Tuan Pham,et al.  Local minima of information-theoretic criteria in blind source separation , 2005, IEEE Signal Processing Letters.

[34]  V. V. Prelov,et al.  Information transmission over channels with additive-multiplicative noise , 1998, Proceedings. 1998 IEEE International Symposium on Information Theory (Cat. No.98CH36252).

[35]  Antonia Maria Tulino,et al.  Monotonic Decrease of the Non-Gaussianness of the Sum of Independent Random Variables: A Simple Proof , 2006, IEEE Transactions on Information Theory.

[36]  Amos Lapidoth,et al.  Capacity bounds via duality with applications to multiple-antenna systems on flat-fading channels , 2003, IEEE Trans. Inf. Theory.

[37]  O. Johnson,et al.  Entropy and Random Vectors , 2001 .

[38]  Shlomo Shamai,et al.  Mutual information and MMSE in gaussian channels , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[39]  I. Vajda.  Theory of statistical inference and information , 1989 .

[40]  Erwin Lutwak,et al.  Information-theoretic inequalities for contoured probability distributions , 2002, IEEE Trans. Inf. Theory.

[41]  Hua Wang,et al.  Vector Gaussian Multiple Description with Individual and Central Receivers , 2006, ISIT.

[42]  Christophe Vignat,et al.  Matrix Fisher inequalities for non-invertible linear systems , 2002, Proceedings IEEE International Symposium on Information Theory,.

[43]  Ram Zamir.  Gaussian codes and Shannon bounds for multiple descriptions , 1999, IEEE Trans. Inf. Theory.

[44]  Tie Liu,et al.  An Extremal Inequality Motivated by Multiterminal Information-Theoretic Problems , 2006, IEEE Transactions on Information Theory.

[45]  Yasutada Oohama.  Gaussian Multiterminal Source Coding with Several Side Informations at the Decoder , 2006, 2006 IEEE International Symposium on Information Theory.

[46]  B H Soffer,et al.  Fisher-based thermodynamics: its Legendre transform and concavity properties. , 1999, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[47]  Christophe Vignat,et al.  AN ENTROPY POWER INEQUALITY FOR THE BINOMIAL FAMILY , 2003 .

[48]  Dinh-Tuan Pham,et al.  Entropy of a variable slightly contaminated with another , 2005, IEEE Signal Processing Letters.

[49]  Sergio Verdú,et al.  Second-order asymptotics of mutual information , 2004, IEEE Transactions on Information Theory.

[50]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[51]  Max H. M. Costa,et al.  On the Gaussian interference channel , 1985, IEEE Trans. Inf. Theory.

[52]  D. A. Bell,et al.  Information Theory and Reliable Communication , 1969 .

[53]  Aaron D. Wyner,et al.  A theorem on the entropy of certain binary sequences and applications-II , 1973, IEEE Trans. Inf. Theory.

[54]  Lang Tong,et al.  Channel estimation under asynchronous packet interference , 2005, IEEE Transactions on Signal Processing.

[55]  Oliver Johnson.  A conditional entropy power inequality for dependent variables , 2004, IEEE Transactions on Information Theory.

[56]  Jacob Binia.  On Divergence-Power Inequalities , 2007, IEEE Transactions on Information Theory.

[57]  Nelson M. Blachman,et al.  The convolution inequality for entropy powers , 1965, IEEE Trans. Inf. Theory.

[58]  Mokshay M. Madiman,et al.  Generalized Entropy Power Inequalities and Monotonicity Properties of Information , 2006, IEEE Transactions on Information Theory.

[59]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[60]  Assaf Naor,et al.  On the rate of convergence in the entropic central limit theorem , 2004 .

[61]  Ram Zamir,et al.  A Proof of the Fisher Information Inequality via a Data Processing Argument , 1998, IEEE Trans. Inf. Theory.

[62]  Edward C. van der Meulen,et al.  Higher-Order Asymptotics of Mutual Information for Nonlinear Channels with Non-Gaussian Noise , 2003, Probl. Inf. Transm..

[63]  R. McEliece,et al.  Some Information Theoretic Saddlepoints , 1985 .

[64]  Ender Tekin,et al.  The Gaussian Multiple Access Wire-Tap Channel , 2006, IEEE Transactions on Information Theory.

[65]  V. Papathanasiou.  Some characteristic properties of the Fisher information matrix via Cacoullos-type inequalities , 1993 .

[66]  Patrick P. Bergmans,et al.  A simple converse for broadcast channels with additive white Gaussian noise (Corresp.) , 1974, IEEE Trans. Inf. Theory.

[67]  D. Donoho.  ON MINIMUM ENTROPY DECONVOLUTION , 1981 .

[68]  E. Lieb.  Proof of an entropy conjecture of Wehrl , 1978 .

[69]  Shlomo Shamai,et al.  A binary analog to the entropy-power inequality , 1990, IEEE Trans. Inf. Theory.

[70]  Daniel Pérez Palomar,et al.  A multivariate generalization of Costa’s entropy power inequality , 2008, 2008 IEEE International Symposium on Information Theory.

[71]  Hiroshi Sato,et al.  An outer bound to the capacity region of broadcast channels (Corresp.) , 1978, IEEE Trans. Inf. Theory.

[72]  Max H. M. Costa,et al.  A new entropy power inequality , 1985, IEEE Trans. Inf. Theory.

[73]  Amiel Feinstein,et al.  Information and information stability of random variables and processes , 1964 .

[74]  Amir Dembo,et al.  Information theoretic inequalities , 1991, IEEE Trans. Inf. Theory.

[75]  Yoshiaki Itoh.  An application of the convolution inequality for the Fisher information , 1989 .

[76]  P. Harremoes,et al.  Entropy and the law of small numbers , 2003, IEEE International Symposium on Information Theory, 2003. Proceedings..

[77]  Hans S. Witsenhausen,et al.  Entropy inequalities for discrete channels , 1974, IEEE Trans. Inf. Theory.

[78]  O. Johnson.  Entropy inequalities and the Central Limit Theorem , 2000 .

[79]  T. Cover,et al.  IEEE TRANSACTIONSON INFORMATIONTHEORY,VOL. IT-30,N0. 6,NOVEmER1984 Correspondence On the Similarity of the Entropy Power Inequality The preceeding equations allow the entropy power inequality and the Brunn-Minkowski Inequality to be rewritten in the equiv , 2022 .

[80]  C. Vignat,et al.  ON FISHER INFORMATION INEQUALITIES AND SCORE FUNCTIONS IN NON-INVERTIBLE LINEAR SYSTEMS , 2003 .

[81]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[82]  Mokshay Madiman,et al.  The Monotonicity of Information in the Central Limit Theorem and Entropy Power Inequalities , 2006, 2006 IEEE International Symposium on Information Theory.

[83]  A. J. Stam.  Some Inequalities Satisfied by the Quantities of Information of Fisher and Shannon , 1959, Inf. Control..

[84]  Amir Dembo,et al.  Simple proof of the concavity of the entropy power with respect to Gaussian noise , 1989, IEEE Trans. Inf. Theory.

[85]  Yasutada Oohama,et al.  The Rate-Distortion Function for the Quadratic Gaussian CEO Problem , 1998, IEEE Trans. Inf. Theory.

[86]  Michel Verleysen,et al.  On the entropy minimization of a linear mixture of variables for source separation , 2005, Signal Process..

[87]  Yasutada Oohama,et al.  Rate-distortion theory for Gaussian multiterminal source coding systems with several side informations at the decoder , 2005, IEEE Transactions on Information Theory.

[88]  Ram Zamir.  A Necessary and Sufficient Condition for Equality in the Matrix Fisher-Information-Inequality , 1997 .

[89]  Edward C. Posner,et al.  Random coding strategies for minimum entropy , 1975, IEEE Trans. Inf. Theory.

[90]  Shlomo Shamai,et al.  Additive non-Gaussian noise channels: mutual information and conditional mean estimation , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[91]  Solomon Kullback,et al.  Information Theory and Statistics , 1970, The Mathematical Gazette.

[92]  O. Johnson.  Information Theory And The Central Limit Theorem , 2004 .

[93]  Sergio Verdú,et al.  Spectral efficiency in the wideband regime , 2002, IEEE Trans. Inf. Theory.

[94]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[95]  Tōkei Sūri Kenkyūjo.  Annals of the Institute of Statistical Mathematics , 1949 .

[96]  E.C. van der Meulen,et al.  Weak signal transmission over certain stationary non-Gaussian channels , 1997, Proceedings of IEEE International Symposium on Information Theory.

[97]  Edward C. van der Meulen,et al.  An asymptotic expression for the information and capacity of a multidimensional channel with weak input signals , 1993, IEEE Trans. Inf. Theory.

[98]  Oliver Johnson,et al.  Concavity of entropy under thinning , 2009, 2009 IEEE International Symposium on Information Theory.

[99]  Hua Wang,et al.  Vector Gaussian Multiple Description With Individual and Central Receivers , 2005, IEEE Transactions on Information Theory.

[100]  Oliver Johnson,et al.  Entropy and the law of small numbers , 2005, IEEE Transactions on Information Theory.

[101]  Daniel Pérez Palomar,et al.  Hessian and Concavity of Mutual Information, Differential Entropy, and Entropy Power in Linear Vector Gaussian Channels , 2009, IEEE Transactions on Information Theory.

[102]  Shlomo Shamai,et al.  A Vector Generalization of Costa's Entropy-Power Inequality With Applications , 2009, IEEE Transactions on Information Theory.

[103]  Shlomo Shamai,et al.  On the achievable throughput of a multiantenna Gaussian broadcast channel , 2003, IEEE Transactions on Information Theory.

[104]  Yasutada Oohama.  Gaussian multiterminal source coding , 1997, IEEE Trans. Inf. Theory.

[105]  Sergio Verdú,et al.  Sensitivity of channel capacity , 1995, IEEE Trans. Inf. Theory.

[106]  Shlomo Shamai,et al.  Proof of Entropy Power Inequalities Via MMSE , 2006, 2006 IEEE International Symposium on Information Theory.

[107]  Aaron D. Wyner,et al.  A theorem on the entropy of certain binary sequences and applications-I , 1973, IEEE Trans. Inf. Theory.