The statistical Minkowski distances: Closed-form formula for Gaussian Mixture Models

The traditional Minkowski distances are induced by the corresponding Minkowski norms in real-valued vector spaces. In this work, we propose novel statistical symmetric distances based on the Minkowski's inequality for probability densities belonging to Lebesgue spaces. These statistical Minkowski distances admit closed-form formula for Gaussian mixture models when parameterized by integer exponents. This result extends to arbitrary mixtures of exponential families with natural parameter spaces being cones: This includes the binomial, the multinomial, the zero-centered Laplacian, the Gaussian and the Wishart mixtures, among others. We also derive a Minkowski's diversity index of a normalized weighted set of probability distributions from Minkowski's inequality.

[1]  M. H. Protter,et al.  THE SOLUTION OF THE PROBLEM OF INTEGRATION IN FINITE TERMS , 1970 .

[2]  Frank Nielsen,et al.  On Hölder Projective Divergences , 2017, Entropy.

[3]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[4]  Frank Nielsen,et al.  A family of statistical symmetric divergences based on Jensen's inequality , 2010, ArXiv.

[5]  Frank Nielsen,et al.  Patch Matching with Polynomial Exponential Families and Projective Divergences , 2016, SISAP.

[6]  Elmer Tolsted,et al.  An Elementary Derivation of the Cauchy, Hölder, and Minkowski Inequalities from Young's Inequality , 1964 .

[7]  S. Eguchi,et al.  Robust parameter estimation with a small bias against heavy contamination , 2008 .

[8]  H. Minkowski,et al.  Geometrie der Zahlen , 1896 .

[9]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[10]  Frank Nielsen,et al.  Bregman vantage point trees for efficient nearest Neighbor Queries , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[11]  Elena Deza,et al.  Encyclopedia of Distances , 2014 .

[12]  Manuel Bronstein,et al.  Symbolic integration I: transcendental functions , 1997 .

[13]  Frank Nielsen,et al.  On The Chain Rule Optimal Transport Distance , 2018, ArXiv.

[14]  Qian Huang,et al.  A new distance measure for probability distribution function of mixture type , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[15]  L. Brown.  Fundamentals of statistical exponential families: with applications in statistical decision theory , 1986 .

[16]  Frank Nielsen,et al.  Generalizing Skew Jensen Divergences and Bregman Divergences With Comparative Convexity , 2017, IEEE Signal Processing Letters.

[17]  Frank Nielsen,et al.  Guaranteed Bounds on Information-Theoretic Measures of Univariate Mixtures Using Piecewise Log-Sum-Exp Inequalities , 2016, Entropy.

[18]  Shun-ichi Amari,et al.  Information Geometry and Its Applications , 2016 .

[19]  C. Alabiso,et al.  A Primer on Hilbert Space Theory: Linear Spaces, Topological Spaces, Metric Spaces, Normed Spaces, and Topological Groups , 2014 .

[20]  Frank Nielsen,et al.  Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means , 2014, Pattern Recognit. Lett..

[21]  Frank Nielsen,et al.  Closed-form information-theoretic divergences for statistical mixtures , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[22]  Frank Nielsen,et al.  On the chi square and higher-order chi distances for approximating f-divergences , 2013, IEEE Signal Processing Letters.

[23]  Amy Shoemaker,et al.  Planar rook algebra with colors and Pascal's simplex , 2012 .

[24]  Frank Nielsen,et al.  The Burbea-Rao and Bhattacharyya Centroids , 2010, IEEE Transactions on Information Theory.

[25]  A. Genz,et al.  Computation of Multivariate Normal and t Probabilities , 2009 .

[26]  Frank Nielsen,et al.  On the Geometry of Mixtures of Prescribed Distributions , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Ling Guan,et al.  Application of Laplacian Mixture Model to Image and Video Retrieval , 2007, IEEE Transactions on Multimedia.

[28]  Baba C. Vemuri,et al.  Robust Point Set Registration Using Gaussian Mixture Models , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  V. K. Balakrishnan,et al.  Introductory discrete mathematics , 1991 .

[30]  Frank Nielsen,et al.  Shape Retrieval Using Hierarchical Total Bregman Soft Clustering , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  David Beymer,et al.  Closed-Form Jensen-Renyi Divergence for Mixture of Gaussians and Applications to Group-Wise Shape Registration , 2009, MICCAI.

[32]  S. Eguchi.  Second Order Efficiency of Minimum Contrast Estimators in a Curved Exponential Family , 1983 .

[33]  Shiri Gordon,et al.  An efficient image similarity measure based on approximations of KL-divergence between two gaussian mixtures , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[34]  L. R. Haff,et al.  Minimax estimation for mixtures of Wishart distributions , 2011, 1203.3342.

[35]  Jean-Philippe Thiran,et al.  Lower and upper bounds for approximation of the Kullback-Leibler divergence between Gaussian Mixture Models , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[36]  D. W. Bolton.  The Multinomial Theorem , 1968 .

[37]  Hagai Aronowitz,et al.  A distance measure between GMMs based on the unscented transform and its application to speaker recognition , 2005, INTERSPEECH.

[38]  Robert Jenssen,et al.  The Cauchy-Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels , 2006, J. Frankl. Inst..

[39]  Inderjit S. Dhillon,et al.  Clustering with Bregman Divergences , 2005, J. Mach. Learn. Res..

[40]  Nancy Bertin,et al.  Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis , 2009, Neural Computation.

[41]  Frank Nielsen,et al.  Statistical exponential families: A digest with flash cards , 2009, ArXiv.

[42]  José Carlos Príncipe,et al.  Closed-form cauchy-schwarz PDF divergence for mixture of Gaussians , 2011, The 2011 International Joint Conference on Neural Networks.