$K$-Means and Gaussian Mixture Modeling with a Separation Constraint

We consider the problem of clustering with $K$-means and Gaussian mixture models with a constraint on the separation between the centers in the context of real-valued data. We first propose a dynamic programming approach to solving the $K$-means problem with a separation constraint on the centers, building on (Wang and Song, 2011). In the context of fitting a Gaussian mixture model, we then propose an EM algorithm that incorporates such a constraint. A separation constraint can help regularize the output of a clustering algorithm, and we provide both simulated and real data examples to illustrate this point.

[1]  Germain Forestier,et al.  Constrained Clustering: Current and New Trends , 2020, A Guided Tour of Artificial Intelligence Research.

[2]  Ning-Zhong Shi,et al.  Likelihood-based approaches for multivariate linear models under inequality constraints for incomplete data , 2012 .

[3]  Ming Tan,et al.  EM-type algorithms for computing restricted MLEs in multivariate normal distributions and multivariate t-distributions , 2008, Comput. Stat. Data Anal..

[4]  S. Ingrassia.  A likelihood-based constrained algorithm for multivariate normal mixture models , 2004 .

[5]  Tibor Szkaliczki,et al.  clustering.sc.dp: Optimal Clustering with Sequential Constraint by Using Dynamic Programming , 2016, R J..

[6]  Ayhan Demiriz,et al.  Constrained K-Means Clustering , 2000 .

[7]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[8]  Salvatore Ingrassia,et al.  Constrained monotone EM algorithms for finite mixture of multivariate Gaussians , 2007, Comput. Stat. Data Anal..

[9]  Salvatore Ingrassia,et al.  Degeneracy of the EM algorithm for the MLE of multivariate Gaussian mixtures and dynamic constraints , 2011, Comput. Stat. Data Anal..

[10]  Dong K. Kim,et al.  The Restricted EM Algorithm for Maximum Likelihood Estimation Under Linear Restrictions on the Parameters , 1995 .

[11]  Xiao-Li Meng,et al.  Maximum likelihood estimation via the ECM algorithm: A general framework , 1993 .

[12]  Arindam Banerjee,et al.  Active Semi-Supervision for Pairwise Constrained Clustering , 2004, SDM.

[13]  Roberto Rocci,et al.  A data driven equivariant approach to constrained Gaussian mixture modeling , 2016, Advances in Data Analysis and Classification.

[14]  Haizhou Wang,et al.  Ckmeans.1d.dp: Optimal k-means Clustering in One Dimension by Dynamic Programming , 2011, R J..

[15]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[16]  N. E. Day.  Estimating the components of a mixture of normal distributions , 1969 .

[17]  P. Deb.  Finite Mixture Models , 2008 .

[18]  R. Hathaway.  A constrained EM algorithm for univariate normal mixtures , 1986 .

[19]  Mortaza Jamshidian,et al.  On Algorithms for Restricted Maximum Likelihood Estimation , 2002, Comput. Stat. Data Anal..

[20]  A FAST EM ALGORITHM FOR QUADRATIC OPTIMIZATION SUBJECT TO CONVEX CONSTRAINTS , 2007 .

[21]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[22]  Taesung Park,et al.  Testing for one-sided group effects in repeated measures study , 2001 .

[23]  Peter J. Rousseeuw,et al.  Finding Groups in Data: An Introduction to Cluster Analysis , 1990 .

[24]  R. Hathaway.  A Constrained Formulation of Maximum-Likelihood Estimation for Normal Mixture Distributions , 1985 .

[25]  Donald Goldfarb,et al.  A numerically stable dual method for solving strictly convex quadratic programs , 1983, Math. Program..

[26]  Joydeep Ghosh,et al.  Scalable Clustering Algorithms with Balancing Constraints , 2006, Data Mining and Knowledge Discovery.

[27]  Mu Qiao,et al.  Gaussian Mixture Models with Component Means Constrained in Pre-selected Subspaces , 2015, ArXiv.

[28]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[29]  Frank Nielsen,et al.  Optimal Interval Clustering: Application to Bregman Clustering and Statistical Mixture Learning , 2014, IEEE Signal Processing Letters.

[30]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[31]  Keiji Takai,et al.  Constrained EM algorithm with projection method , 2011, Computational Statistics.

[32]  D. Goldfarb,et al.  Dual and primal-dual methods for solving strictly convex quadratic programs , 1982 .

[33]  Wei Pan,et al.  Penalized Model-Based Clustering with Application to Variable Selection , 2007, J. Mach. Learn. Res..

[34]  Ian Davidson,et al.  Reveling in Constraints , 2009, ACM Queue.

[35]  Zheng Shurong,et al.  The restricted EM algorithm under linear inequalities in a linear model with missing data , 2005 .

[36]  Ning-Zhong Shi,et al.  The restricted EM algorithm under inequality restrictions on the parameters , 2005 .

[37]  Claire Cardie,et al.  Proceedings of the Eighteenth International Conference on Machine Learning, 2001, p. 577–584. Constrained K-means Clustering with Background Knowledge , 2022 .

[38]  ECM and MM algorithms for normal mixtures with constrained parameters , 2013 .

[39]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[40]  S. S. Ravi,et al.  Clustering with Constraints: Feasibility Issues and the k-Means Algorithm , 2005, SDM.

[41]  Ian Davidson,et al.  Constrained Clustering: Advances in Algorithms, Theory, and Applications , 2008 .

[42]  Ji Zhu,et al.  Variable Selection for Model‐Based High‐Dimensional Clustering and Its Application to Microarray Data , 2008, Biometrics.

[43]  S. Sinha,et al.  Testing for generalized linear mixed models with cluster correlated data under linear inequality constraints , 2012 .