Finite mixture biclustering of discrete type multivariate data

Many of the methods which deal with clustering in matrices of data are based on mathematical techniques such as distance-based algorithms or matrix decomposition and eigenvalues. In general, it is not possible to use statistical inferences or select the appropriateness of a model via information criteria with these techniques because there is no underlying probability model. This article summarizes some recent model-based methodologies for matrices of binary, count, and ordinal data, which are modelled under a unified statistical framework using finite mixtures to group the rows and/or columns. The model parameter can be constructed from a linear predictor of parameters and covariates through link functions. This likelihood-based one-mode and two-mode fuzzy clustering provides maximum likelihood estimation of parameters and the options of using likelihood information criteria for model comparison. Additionally, a Bayesian approach is presented in which the parameters and the number of clusters are estimated simultaneously from their joint posterior distribution. Visualization tools focused on ordinal data, the fuzziness of the clustering structures, and analogies of various standard plots used in the multivariate analysis are presented. Finally, a set of future extensions is enumerated.

[1]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[2]  Daniel Ståhl,et al.  Model‐based cluster analysis , 2012 .

[3]  Adrian E. Raftery,et al.  How Many Clusters? Which Clustering Method? Answers Via Model-Based Cluster Analysis , 1998, Comput. J..

[4]  G. J. McLachlan,et al.  9 The classification and mixture maximum likelihood approaches to cluster analysis , 1982, Classification, Pattern Recognition and Reduction of Dimensionality.

[5]  C. Robert,et al.  Estimating Mixtures of Regressions , 2003 .

[6]  P. McCullagh,et al.  How many clusters , 2008 .

[7]  B. Everitt,et al.  Cluster Analysis: Everitt/Cluster Analysis , 2011 .

[8]  Margarida G. M. S. Cardoso,et al.  Identifying the number of clusters in discrete mixture models , 2014, 1409.7419.

[9]  G. Quinn,et al.  Experimental Design and Data Analysis for Biologists , 2002 .

[10]  J. A. Hartigan,et al.  Mosaics for Contingency Tables , 1981 .

[11]  K. Liang,et al.  Asymptotic Properties of Maximum Likelihood Estimators and Likelihood Ratio Tests under Nonstandard Conditions , 1987 .

[12]  Sara Taskinen,et al.  Model‐based approaches to unconstrained ordination , 2015 .

[13]  Maurizio Vichi,et al.  Two-mode multi-partitioning , 2008, Comput. Stat. Data Anal..

[14]  Volodymyr Melnykov,et al.  Finite mixture models and model-based clustering , 2010 .

[15]  S. Pledger.  Unified Maximum Likelihood Estimates for Closed Capture–Recapture Models Using Mixtures , 2000, Biometrics.

[16]  S. Frühwirth-Schnatter.  Markov chain Monte Carlo Estimation of Classical and Dynamic Switching and Mixture Models , 2001 .

[17]  Gérard Govaert,et al.  An EM algorithm for the block mixture model , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  B. McCune,et al.  Analysis of Ecological Communities , 2002 .

[19]  M. Stephens.  Dealing with label switching in mixture models , 2000 .

[20]  Volodymyr Melnykov,et al.  Finite mixture modelling in mass spectrometry analysis , 2013 .

[21]  Yu Hayakawa,et al.  Capture–Recapture Estimation Using Finite Mixtures of Arbitrary Dimension , 2010, Biometrics.

[22]  Tom Lodewyckx,et al.  Bayesian Versus Frequentist Inference , 2008 .

[23]  Julien Jacques,et al.  Simultaneous Clustering and Model Selection for Multinomial Distribution: A Comparative Study , 2015, IDA.

[24]  Geoffrey J. McLachlan,et al.  Mixture models : inference and applications to clustering , 1989 .

[25]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[26]  Gérard Govaert,et al.  A Comparison Between Block CEM and Two-Way CEM Algorithms to Cluster a Contingency Table , 2005, PKDD.

[27]  A. Agresti.  Analysis of Ordinal Categorical Data , 1985 .

[28]  D. Fernández,et al.  Mixture-based clustering for the ordered stereotype model , 2016, Comput. Stat. Data Anal..

[29]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[30]  Catherine A. Sugar,et al.  Finding the Number of Clusters in a Dataset , 2003 .

[31]  A. Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[32]  Ajay Jasra,et al.  Markov Chain Monte Carlo Methods and the Label Switching Problem in Bayesian Mixture Modeling , 2005 .

[33]  Ivy Liu,et al.  Biclustering Models for Two-Mode Ordinal Data , 2016, Psychometrika.

[34]  Christian P. Robert,et al.  Reversible Jump MCMC Converging to Birth-and-Death MCMC and More General Continuous Time Samplers , 2001 .

[35]  Jeroen K. Vermunt,et al.  The Use of Restricted Latent Class Models for Defining and Testing Nonparametric and Parametric Item Response Theory Models , 2001 .

[36]  Richard Breen,et al.  Mixture Models for Ordinal Data , 2010 .

[37]  Daniel Fernández,et al.  Categorising Count Data into Ordinal Responses with Application to Ecological Communities , 2016 .

[38]  Rebecca A Betensky,et al.  A penalized latent class model for ordinal data. , 2007, Biostatistics.

[39]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[40]  Gérard Govaert,et al.  Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  J. A. Hartigan,et al.  A k-means clustering algorithm , 1979 .

[42]  B. Manly.  Multivariate Statistical Methods : A Primer , 1986 .

[43]  Francesco Bartolucci,et al.  Longitudinal analysis of self‐reported health status by mixture latent auto‐regressive models , 2014 .

[44]  A. Agresti.  Analysis of Ordinal Categorical Data: Agresti/Analysis , 2010 .

[45]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[46]  Damien McParland,et al.  Clustering Ordinal Data via Latent Variable Models , 2013, Algorithms from and for Nature and Life.

[47]  Philip S. Yu,et al.  Top 10 algorithms in data mining , 2007, Knowledge and Information Systems.

[48]  Jean-Michel Marin,et al.  Bayesian Core: A Practical Approach to Computational Bayesian Statistics , 2010 .

[49]  Marco Alfò,et al.  Advances in Mixture Models , 2007, Comput. Stat. Data Anal..

[50]  P. Green,et al.  Corrigendum: On Bayesian analysis of mixtures with an unknown number of components , 1997 .

[51]  Gilles Celeux,et al.  Bayesian Inference for Mixture: The Label Switching Problem , 1998, COMPSTAT.

[52]  Ryan P. Browne,et al.  Model-based clustering, classification, and discriminant analysis of data with mixed type , 2012 .

[53]  H. P. Friedman,et al.  On Some Invariant Criteria for Grouping Data , 1967 .

[54]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[55]  Gérard Govaert,et al.  Clustering with block mixture models , 2003, Pattern Recognit..

[56]  S. Frühwirth-Schnatter,et al.  Model-based clustering of categorical time series , 2010 .

[57]  A. Raftery,et al.  Variable Selection for Model-Based Clustering , 2006 .

[58]  Richard Arnold,et al.  Multivariate methods using mixtures: Correspondence analysis, scaling and pattern-detection , 2014, Comput. Stat. Data Anal..

[59]  O. Cappé,et al.  Reversible jump, birth‐and‐death and more general continuous time Markov chain Monte Carlo samplers , 2003 .

[60]  Damien McParland,et al.  Model based clustering for mixed data: clustMD , 2015, Advances in Data Analysis and Classification.

[61]  F. Kianifard.  Applied Multivariate Data Analysis: Volume II: Categorical and Multivariate Methods , 1994 .

[62]  Irini Moustaki,et al.  A Latent Variable Model for Ordinal Variables , 2000 .

[63]  M. Stephens.  Bayesian analysis of mixture models with an unknown number of components- an alternative to reversible jump methods , 2000 .

[64]  Sylvia Frühwirth-Schnatter,et al.  Finite Mixture and Markov Switching Models , 2006 .

[65]  S. C. Johnson.  Hierarchical clustering schemes , 1967, Psychometrika.

[66]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[67]  Halina Frydman.  Estimation in the Mixture of Markov Chains Moving With Different Speeds , 2003 .

[68]  L. A. Goodman.  Exploratory latent structure analysis using both identifiable and unidentifiable models , 1974 .

[69]  Alan D. Marrs.  An Application of Reversible-Jump MCMC to Multivariate Spherical Gaussian Mixtures , 1997, NIPS.

[70]  S. Frühwirth-Schnatter,et al.  Labor Market Entry and Earnings Dynamics: Bayesian Inference Using Mixtures-of-Experts Markov Chain Clustering , 2012 .

[71]  Richard Paap,et al.  A Bayesian approach to two-mode clustering , 2009 .

[72]  A. McCutcheon,et al.  Latent Class Analysis , 2021, Encyclopedia of Autism Spectrum Disorders.

[73]  A Agresti,et al.  Quasi-symmetric latent class models, with application to rater agreement. , 1993, Biometrics.

[74]  Robert Tibshirani,et al.  Cluster Validation by Prediction Strength , 2005 .

[75]  J. Hagenaars,et al.  Methods in Human Growth Research: Ordinal longitudinal data analysis , 2004 .

[76]  Daniel Fernández,et al.  A goodness-of-fit test for the ordered stereotype model. , 2016, Statistics in medicine.

[77]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[78]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[79]  P. Diggle,et al.  Analysis of Longitudinal Data , 2003 .

[80]  J. Anderson.  Regression and Ordered Categorical Variables , 1984 .

[81]  Petros Dellaportas,et al.  Multivariate mixtures of normals with unknown number of components , 2006, Stat. Comput..

[82]  Xin-Yuan Song,et al.  A mixture of generalized latent variable models for mixed mode and heterogeneous data , 2011, Comput. Stat. Data Anal..

[83]  B. Manly.  Randomization, Bootstrap and Monte Carlo Methods in Biology , 2018 .

[84]  G. McLachlan.  On Bootstrapping the Likelihood Ratio Test Statistic for the Number of Components in a Normal Mixture , 1987 .

[85]  Vichi Maurizio.  Double k-means Clustering for Simultaneous Classification of Objects and Variables , 2001 .

[86]  Jeroen K. Vermunt,et al.  A nonparametric random-coefficients approach : The latest class regression model , 2001 .

[87]  Brian Everitt,et al.  Cluster analysis , 1974 .

[88]  Zhihua Zhang,et al.  Learning a multivariate Gaussian mixture model with the reversible jump MCMC algorithm , 2004, Stat. Comput..

[89]  N. Gotelli,et al.  NULL MODELS IN ECOLOGY , 1996 .

[90]  P. McCullagh.  Regression Models for Ordinal Data , 1980 .

[91]  P. Green,et al.  On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion) , 1997 .

[92]  R. Brant.  Assessing proportionality in the proportional odds model for ordinal logistic regression. , 1990, Biometrics.

[93]  Nial Friel,et al.  Block clustering with collapsed latent block models , 2010, Stat. Comput..

[94]  D. Fernández,et al.  Model selection for mixture‐based clustering for ordinal data , 2016 .

[95]  Jean-Michel Marin,et al.  Bayesian Inference on Mixtures of Distributions , 2008, 0804.2413.

[96]  S. Haberman.  Analysis of qualitative data , 1978 .

[97]  G. Govaert,et al.  Latent Block Model for Contingency Table , 2010 .

[98]  Adrian E. Raftery,et al.  Bayesian Regularization for Normal Mixture Estimation and Model-Based Clustering , 2007, J. Classif..