φ-DIVERGENCES , SUFFICIENCY , BAYES SUFFICIENCY , AND DEFICIENCY

The paper studies the relations between φ-divergences and fundamental concepts of decision theory such as sufficiency, Bayes sufficiency, and LeCam’s deficiency. A new and considerably simplified approach is given to the spectral representation of φ-divergences already established in Österreicher and Feldman [28] under restrictive conditions and in Liese and Vajda [22], [23] in the general form. The simplification is achieved by a new integral representation of convex functions in terms of elementary convex functions which are strictly convex at one point only. Bayes sufficiency is characterized with the help of a binary model that consists of the joint distribution and the product of the marginal distributions of the observation and the parameter, respectively. LeCam’s deficiency is expressed in terms of φ-divergences where φ belongs to a class of convex functions whose curvature measures are finite and satisfy a normalization condition.

[1]  Timothy R. C. Read,et al.  Goodness-Of-Fit Statistics for Discrete Multivariate Data , 1988 .

[2]  I. Vincze.  On the Concept and Measure of Information Contained in an Observation , 1981 .

[3]  Igor Vajda,et al.  On convergence of information contained in quantized observations , 2002, IEEE Trans. Inf. Theory.

[4]  E. Torgersen.  Comparison of Statistical Experiments , 1991 .

[5]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[6]  F. Österreicher,et al.  Divergenzen von Wahrscheinlichkeitsverteilungen — Integralgeometrisch Betrachtet , 1981 .

[7]  Flemming Topsøe,et al.  Information-theoretical optimization techniques , 1979, Kybernetika.

[8]  Martin J. Wainwright,et al.  ON surrogate loss functions and f-divergences , 2005, math/0510521.

[9]  Flemming Topsøe,et al.  Some inequalities for information divergence and related measures of discrimination , 2000, IEEE Trans. Inf. Theory.

[10]  David Lindley,et al.  Optimal Statistical Decisions , 1971 .

[11]  Igor Vajda,et al.  On Divergences and Informations in Statistics and Information Theory , 2006, IEEE Transactions on Information Theory.

[12]  T. Kailath.  The Divergence and Bhattacharyya Distance Measures in Signal Selection , 1967 .

[13]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[14]  Ferdinand Österreicher,et al.  Statistical information and discrimination , 1993, IEEE Trans. Inf. Theory.

[15]  Edward C. van der Meulen,et al.  About the Asymptotic Accuracy of Barron Density Estimates , 1998, IEEE Trans. Inf. Theory.

[16]  L. L. Cam,et al.  Asymptotic Methods In Statistical Decision Theory , 1986 .

[17]  H. Vincent Poor,et al.  Robust decision design using a distance criterion , 1980, IEEE Trans. Inf. Theory.

[18]  László Györfi,et al.  Distribution estimation consistent in total variation and in two types of information divergence , 1992, IEEE Trans. Inf. Theory.

[19]  Andrew R. Barron,et al.  Information-theoretic asymptotics of Bayes methods , 1990, IEEE Trans. Inf. Theory.

[20]  H. Chernoff.  A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations , 1952 .

[21]  I. Vajda.  Theory of statistical inference and information , 1989 .

[22]  A characterization of sufficiency by power functions , 1974 .

[23]  Le Cam,et al.  Locally asymptotically normal families of distributions : certain approximations to families of distributions & thier use in the theory of estimation & testing hypotheses , 1960 .

[24]  K. Matusita.  Decision Rules, Based on the Distance, for Problems of Fit, Two Samples, and Estimation , 1955 .

[25]  Igor Vajda,et al.  On Metric Divergences of Probability Measures , 2009, Kybernetika.

[26]  I. Vajda.  On thef-divergence and singularity of probability measures , 1972 .

[27]  J. Wellner,et al.  GOODNESS-OF-FIT TESTS VIA PHI-DIVERGENCES , 2006, math/0603238.

[28]  Suguru Arimoto,et al.  Information-Theoretical Considerations on Estimation Problems , 1971, Inf. Control..

[29]  S. M. Ali,et al.  A General Class of Coefficients of Divergence of One Distribution from Another , 1966 .

[30]  Aditya Guntuboyina.  Lower Bounds for the Minimax Risk Using $f$-Divergences, and Applications , 2011, IEEE Transactions on Information Theory.

[31]  S. Kakutani.  On Equivalence of Infinite Product Measures , 1948 .

[32]  Klaus J. Miescke,et al.  Statistical decision theory : estimation, testing, and selection , 2008 .