Closed-form information-theoretic divergences for statistical mixtures

Statistical mixtures such as Rayleigh, Wishart or Gaussian mixture models are commonly used in pattern recognition and signal processing tasks. Since the Kullback-Leibler divergence between any two such mixture models does not admit an analytical expression, the relative entropy can only be approximated numerically using time-consuming Monte-Carlo stochastic sampling. This drawback has motivated the quest for alternative information-theoretic divergences such as the recent Jensen-Rényi, Cauchy-Schwarz, or total square loss divergences that bypass the numerical approximations by providing exact analytic expressions. In this paper, we state sufficient conditions on the mixture distribution family so that these novel non-KL statistical divergences between any two such mixtures can be expressed in generic closed-form formulas.

[1]  John R. Hershey,et al.  Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[2]  David Beymer,et al.  Closed-Form Jensen-Renyi Divergence for Mixture of Gaussians and Applications to Group-Wise Shape Registration , 2009, MICCAI.

[3]  Ling Guan,et al.  Application of Laplacian Mixture Model to Image and Video Retrieval , 2007, IEEE Transactions on Multimedia.

[4]  L. R. Haff,et al.  Minimax estimation for mixtures of Wishart distributions , 2011, 1203.3342.

[5]  Jonathan M. Nichols,et al.  Calculation of Differential Entropy for a Mixed Gaussian Distribution , 2008, Entropy.

[6]  Yali Amit,et al.  Generative Models for Labeling Multi-object Configurations in Images , 2006, Toward Category-Level Object Recognition.

[7]  Petia Radeva,et al.  Rayleigh Mixture Model for Plaque Characterization in Intravascular Ultrasound , 2011, IEEE Transactions on Biomedical Engineering.

[8]  Vassilios Morellas,et al.  Tensor Sparse Coding for Region Covariances , 2010, ECCV.

[9]  Inderjit S. Dhillon,et al.  Clustering with Bregman Divergences , 2005, J. Mach. Learn. Res..

[10]  John R. Hershey,et al.  Accelerated Monte Carlo for Kullback-Leibler divergence between Gaussian mixture models , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  Robert Jenssen,et al.  The Cauchy-Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels , 2006, J. Frankl. Inst..

[12]  Lawrence D. Brown.  Fundamentals of Statistical Exponential Families , 1987 .

[13]  Frank Nielsen,et al.  Shape Retrieval Using Hierarchical Total Bregman Soft Clustering , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  José Carlos Príncipe,et al.  Closed-form cauchy-schwarz PDF divergence for mixture of Gaussians , 2011, The 2011 International Joint Conference on Neural Networks.

[15]  Sullivan Hidot,et al.  An Expectation-Maximization algorithm for the Wishart mixture model: Application to movement clustering , 2010, Pattern Recognit. Lett..

[16]  Frank Nielsen,et al.  K-MLE: A fast algorithm for learning statistical mixture models , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).