Generalised Pinsker Inequalities

We generalise the classical Pinsker inequality which relates variational divergence to Kullback-Liebler divergence in two ways: we consider arbitrary f-divergences in place of KL divergence, and we assume knowledge of a sequence of values of generalised variational divergences. We then develop a best possible inequality for this doubly generalised situation. Specialising our result to the classical case provides a new and tight explicit bound relating KL to variational divergence (solving a problem posed by Vajda some 40 years ago). The solution relies on exploiting a connection between divergences and the Bayes risk of a learning problem via an integral representation.

[1]  Alison L Gibbs,et al.  On Choosing and Bounding Probability Metrics , 2002, math/0209021.

[2]  Gustavo L. Gilardoni.  On the minimum f-divergence for given total variation , 2006 .

[3]  T. Aaron Gulliver,et al.  Confliction of the Convexity and Metric Properties in f-Divergences , 2007, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[4]  G. Toussaint.  Probability of error, expected divergence, and the affinity of several distributions , 1978 .

[5]  Inder Jeet Taneja,et al.  Inequalities Among Symmetric Divergence Measures and Their Refinement , 2005 .

[6]  Amiel Feinstein,et al.  Information and information stability of random variables and processes , 1964 .

[7]  M. Degroot.  Uncertainty, Information, and Sequential Experiments , 1962 .

[8]  Solomon Kullback,et al.  Correction to A Lower Bound for Discrimination Information in Terms of Variation , 1970, IEEE Trans. Inf. Theory.

[9]  C. Villani,et al.  Weighted Csiszár-Kullback-Pinsker inequalities and applications to transportation inequalities , 2005 .

[10]  L. L. Cam,et al.  Asymptotic Methods In Statistical Decision Theory , 1986 .

[11]  S. Dragomir,et al.  Csiszár f-divergence, Ostrowski’s inequality and mutual information , 2001 .

[12]  Flemming Topsøe,et al.  Some inequalities for information divergence and related measures of discrimination , 2000, IEEE Trans. Inf. Theory.

[13]  Gustavo L. Gilardoni.  On Pinsker's and Vajda's Type Inequalities for Csiszár's $f$ -Divergences , 2006, IEEE Transactions on Information Theory.

[14]  S. Kullback,et al.  A lower bound for discrimination information in terms of variation (Corresp.) , 1967, IEEE Trans. Inf. Theory.

[15]  Mark D. Reid,et al.  Information, Divergence and Risk for Binary Experiments , 2009, J. Mach. Learn. Res..

[16]  S. M. Ali,et al.  A General Class of Coefficients of Divergence of One Distribution from Another , 1966 .

[17]  Inder Jeet Taneja.  Bounds on Non - Symmetric Divergence Measures in terms of Symmetric Divergence Measures - communica , 2004 .

[18]  Michael I. Jordan,et al.  On distance measures, surrogate loss functions, and distributed detection , 2005 .

[19]  Igor Vajda,et al.  On Divergences and Informations in Statistics and Information Theory , 2006, IEEE Transactions on Information Theory.

[20]  A. Arnold,et al.  On generalized Csiszár-Kullback inequalities , 2000 .

[21]  F. Topsøe.  BOUNDS FOR ENTROPY AND DIVERGENCE FOR DISTRIBUTIONS OVER A TWO-ELEMENT SET , 2001 .

[22]  Ferdinand Österreicher,et al.  Statistical information and discrimination , 1993, IEEE Trans. Inf. Theory.

[23]  Lang P. Withers,et al.  Some inequalities relating different measures of divergence between two probability distributions , 1999, IEEE Trans. Inf. Theory.

[24]  Pranesh Kumar,et al.  A symmetric information divergence measure of the Csiszár's f-divergence class and its bounds , 2005 .

[25]  Peter Harremoës,et al.  Refinements of Pinsker's inequality , 2003, IEEE Trans. Inf. Theory.

[26]  I. J. Taneja.  REFINEMENT INEQUALITIES AMONG SYMMETRIC DIVERGENCE MEASURES , 2005 .