LEEM: Lean Elastic EM for Gaussian Mixture Model via Bounds-Based Filtering

Gaussian Mixture Model (GMM) is widely used in characterizing complicated real-world data and has played a crucial role in many pattern recognition problems. GMM is usually trained by Expectation Maximization algorithm (EM) which is computationally intensive. Previous studies have proposed a family of variants of EM. By considering only the data points that are the most important to a model in a GMM when updating that model, they help reduce some GMM training time. They are named Elastic EM in this paper. This work proposes several novel optimizations to further accelerate Elastic EM. These optimizations detect and avoid unnecessary probability calculations through novel bounds-based filtering at E-step as well as a Delta optimization to the M-step. Together, they create Lean Elastic EM (LEEM), which brings multi-fold speedups on six datasets of various sizes and dimensions.

[1]  Marc A. Zissman,et al.  Automatic language identification using Gaussian mixture and hidden Markov models , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[2]  Zoran Zivkovic,et al.  Improved adaptive Gaussian mixture model for background subtraction , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[3]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[4]  Shashidhar G. Koolagudi,et al.  Real Life Emotion Classification using Spectral Features and Gaussian Mixture Models , 2012 .

[5]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[6]  Bo Thiesson,et al.  Accelerating EM for Large Databases , 2001, Machine Learning.

[7]  Xipeng Shen,et al.  FALCON: A Fast Drop-In Replacement of Citation KNN for Multiple Instance Learning , 2018, CIKM.

[8]  Andrew McCallum,et al.  Efficient clustering of high-dimensional data sets with application to reference matching , 2000, KDD '00.

[9]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[10]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[11]  Santanu Chaudhury,et al.  Efficient Skin Region Segmentation Using Low Complexity Fuzzy Decision Tree Model , 2009, 2009 Annual IEEE India Conference.

[12]  H. Akaike.  A new look at the statistical model identification , 1974 .

[13]  Georg Heigold,et al.  GMM-free DNN acoustic model training , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Allou Samé,et al.  An online classification EM algorithm based on the mixture model , 2007, Stat. Comput..

[15]  Douglas A. Reynolds,et al.  Dialect identification using Gaussian mixture models , 2004, Odyssey.

[16]  Yue Zhao,et al.  Yinyang K-Means: A Drop-In Replacement of the Classic K-Means with Consistent Speedup , 2015, ICML.

[17]  Douglas A. Reynolds,et al.  Robust text-independent speaker identification using Gaussian mixture speaker models , 1995, IEEE Trans. Speech Audio Process..

[18]  R. Jennrich,et al.  Conjugate Gradient Acceleration of the EM Algorithm , 1993 .

[19]  PoggioTomaso,et al.  Example-Based Learning for View-Based Human Face Detection , 1998 .

[20]  M. Nadif,et al.  Speed-up for the expectation-maximization algorithm for clustering categorical data , 2007, J. Glob. Optim..

[21]  I-Cheng Yeh,et al.  The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients , 2009, Expert Syst. Appl..

[22]  Guoyang Chen,et al.  Sweet KNN: An Efficient KNN on GPU through Reconciliation between Redundancy Removal and Regularity , 2017, 2017 IEEE 33rd International Conference on Data Engineering (ICDE).

[23]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[24]  Qiang Chen,et al.  Fuzzy Local Gaussian Mixture Model for Brain MR Image Segmentation , 2012, IEEE Transactions on Information Technology in Biomedicine.

[25]  Allou Samé,et al.  A classification EM algorithm for binned data , 2006, Comput. Stat. Data Anal..

[26]  Haim H. Permuter,et al.  A study of Gaussian mixture models of color and texture features for image classification and segmentation , 2006, Pattern Recognit..

[27]  G. Celeux,et al.  A Classification EM algorithm for clustering and two stochastic versions , 1992 .

[28]  Denis J. Dean,et al.  Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables , 1999 .