On the Maximum Values of f-Divergence and Rényi Divergence under a Given Variational Distance

We consider the problem of finding maximum values of f-divergences Df(P ∥ Q) of discrete probability distributions P and Q with values on a finite set under the condition that the variation distance V(P, Q) between them and one of the distributions P or Q are given. We obtain exact expressions for such maxima of f-divergences, which in a number of cases allow to obtain both explicit formulas and upper bounds for them. As a consequence, we obtain explicit expressions for the maxima of f-divergences Df (P ∥ Q) given that, besides V(P, Q), we only know the value of the maximum component of either P or Q. Analogous results are also obtained for the Renyi divergence.

[1]  Sergio Verdú,et al.  Upper bounds on the relative entropy and Rényi divergence as a function of total variation distance for finite alphabets , 2015, 2015 IEEE Information Theory Workshop - Fall (ITW).

[2]  Gustavo L. Gilardoni.  On the minimum f-divergence for given total variation , 2006 .

[3]  J. Aczel,et al.  On Measures of Information and Their Characterizations , 2012 .

[4]  Sergio Verdú,et al.  $f$ -Divergence Inequalities , 2015, IEEE Transactions on Information Theory.

[5]  Igor Vajda,et al.  On Divergences and Informations in Statistics and Information Theory , 2006, IEEE Transactions on Information Theory.

[6]  Vyacheslav V. Prelov.  Optimal Upper Bounds for the Divergence of Finite-Dimensional Distributions under a Given Variational Distance , 2019, Probl. Inf. Transm..

[7]  Peter Harremoës,et al.  Rényi Divergence and Kullback-Leibler Divergence , 2012, IEEE Transactions on Information Theory.

[8]  Adityanand Guntuboyina,et al.  Sharp Inequalities for $f$ -Divergences , 2014, IEEE Trans. Inf. Theory.

[9]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[10]  Igal Sason.  Tight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding , 2015, IEEE Transactions on Information Theory.

[11]  Gustavo L. Gilardoni.  On Pinsker's and Vajda's Type Inequalities for Csiszár's $f$ -Divergences , 2006, IEEE Transactions on Information Theory.

[12]  Jacob Ziv,et al.  On functionals satisfying a data-processing theorem , 1973, IEEE Trans. Inf. Theory.

[13]  S. M. Ali,et al.  A General Class of Coefficients of Divergence of One Distribution from Another , 1966 .