Clustering, classification, discriminant analysis, and dimension reduction via generalized hyperbolic mixtures

A method for dimension reduction with clustering, classification, or discriminant analysis is introduced. This mixture model-based approach is based on fitting generalized hyperbolic mixtures on a reduced subspace within the paradigm of model-based clustering, classification, or discriminant analysis. A reduced subspace of the data is derived by considering the extent to which group means and group covariances vary. The members of the subspace arise through linear combinations of the original data, and are ordered by importance via the associated eigenvalues. The observations can be projected onto the subspace, resulting in a set of variables that captures most of the clustering information available. The use of generalized hyperbolic mixtures gives a robust framework capable of dealing with skewed clusters. Although dimension reduction is increasingly in demand across various application areas, many applications are biological and so some of the real data examples are within that sphere. Simulated data are also used for illustration.

[1]  Paul D. McNicholas,et al.  Model-based clustering of microarray expression data via latent Gaussian mixture models , 2010, Bioinform..

[2]  Adrian E. Raftery,et al.  Normal Mixture Modelling for Model-Based Clustering,Classification, and Density Estimation , 2015 .

[3]  Xianggui Qu,et al.  Multivariate Data Analysis , 2007, Technometrics.

[4]  O. Barndorff-Nielsen.  Exponentially decreasing distributions for the logarithm of particle size , 1977, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences.

[5]  Paul D. McNicholas,et al.  On Clustering and Classification Via Mixtures of Multivariate t-Distributions , 2013, Statistical Models for Data Analysis.

[6]  Geoffrey J. McLachlan,et al.  Robust mixture modelling using the t distribution , 2000, Stat. Comput..

[7]  Ryan P. Browne,et al.  Mixtures of Variance-Gamma Distributions , 2013, 1309.2695.

[8]  Paul D. McNicholas,et al.  Serial and parallel implementations of model-based clustering via parsimonious Gaussian mixture models , 2010, Comput. Stat. Data Anal..

[9]  W. Härdle,et al.  Applied Multivariate Statistical Analysis , 2003 .

[10]  Salvatore Ingrassia,et al.  Constrained monotone EM algorithms for mixtures of multivariate t distributions , 2010, Stat. Comput..

[11]  P. McNicholas,et al.  Mixtures of modified t-factor analyzers for model-based clustering, classification, and discriminant , 2011 .

[12]  Mia Hubert,et al.  ROBPCA: A New Approach to Robust Principal Component Analysis , 2005, Technometrics.

[13]  Irene Vrbik,et al.  Analytic calculations for the EM algorithm for multivariate skew-t mixture models , 2012 .

[14]  Gérard Govaert,et al.  Gaussian parsimonious clustering models , 1995, Pattern Recognit..

[15]  Bernard D. Flury,et al.  Why Multivariate Statistics , 1997 .

[16]  Ryan P. Browne,et al.  A mixture of common skew‐t factor analysers , 2013, 1307.5558.

[17]  Paul D. McNicholas,et al.  Capturing patterns via parsimonious t mixture models , 2013, 1303.2316.

[18]  R. Tibshirani,et al.  Discriminant Analysis by Gaussian Mixtures , 1996 .

[19]  A. Raftery,et al.  Model-based Gaussian and non-Gaussian clustering , 1993 .

[20]  R. Tibshirani,et al.  Significance analysis of microarrays applied to the ionizing radiation response , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[21]  Paul D. McNicholas,et al.  Fractionally-Supervised Classification , 2013, Journal of Classification.

[22]  Michelle A. Steane,et al.  Model-Based Classification via Mixtures of Multivariate t-Factor Analyzers , 2012, Commun. Stat. Simul. Comput..

[23]  B. Lindsay.  Mixture models : theory, geometry, and applications , 1995 .

[24]  Tsung I. Lin,et al.  Robust mixture modeling using multivariate skew t distributions , 2010, Stat. Comput..

[25]  Paul D. McNicholas,et al.  Model-based classification via mixtures of multivariate t-distributions , 2011, Comput. Stat. Data Anal..

[26]  H. Riedwyl,et al.  Multivariate Statistics: A Practical Approach , 1988 .

[27]  Paul D. McNicholas,et al.  Model-based clustering, classification, and discriminant analysis via mixtures of multivariate t-distributions , 2011, Statistics and Computing.

[28]  L. Hubert,et al.  Comparing partitions , 1985 .

[29]  Luca Scrucca,et al.  Graphical tools for model-based mixture discriminant analysis , 2013, Advances in Data Analysis and Classification.

[30]  D. N. Geary.  Mixture Models: Inference and Applications to Clustering , 1989 .

[31]  Paul D. McNicholas,et al.  Parsimonious Gaussian mixture models , 2008, Stat. Comput..

[32]  Ryan P. Browne,et al.  A mixture of generalized hyperbolic factor analyzers , 2013, Advances in Data Analysis and Classification.

[33]  U. Alon,et al.  Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[34]  Ker-Chau Li.  Sliced inverse regression for dimension reduction (with discussion) , 1991 .

[35]  N. Dean,et al.  Using unlabelled data to update classification rules with applications in food authenticity studies , 2006 .

[36]  Geoffrey J. McLachlan,et al.  Mixtures of Factor Analyzers with Common Factor Loadings: Applications to the Clustering and Visualization of High-Dimensional Data , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Adrian E. Raftery,et al.  MCLUST: Software for Model-Based Cluster Analysis , 1999 .

[38]  Paul D. McNicholas,et al.  Dimension reduction for model-based clustering via mixtures of multivariate $$t$$t-distributions , 2013, Adv. Data Anal. Classif..

[39]  Ryan P. Browne,et al.  Unsupervised learning via mixtures of skewed distributions with hypercube contours , 2014, Pattern Recognit. Lett..

[40]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[41]  Ker-Chau Li,et al.  Sliced Inverse Regression for Dimension Reduction , 1991 .

[42]  P. McNicholas.  Model-based classification using latent Gaussian mixture models , 2010 .

[43]  Ryan P. Browne,et al.  Mixtures of multivariate power exponential distributions , 2015, Biometrics.

[44]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .

[45]  A. C. Aitken.  XXV.—On Bernoulli's Numerical Solution of Algebraic Equations , 1927 .

[46]  Ryan P. Browne,et al.  A mixture of generalized hyperbolic distributions , 2013, 1305.1036.

[47]  King C. P. Li.  High dimensional data analysis via the sir/phd approach , 2000 .

[48]  Geoffrey J. McLachlan,et al.  Finite mixtures of multivariate skew t-distributions: some recent and new results , 2014, Stat. Comput..

[49]  Adrian E. Raftery,et al.  clustvarsel: A Package Implementing Variable Selection for Model-based Clustering in R , 2014, 1411.0606.

[50]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[51]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[52]  Paul D. McNicholas,et al.  Dimension reduction for model-based clustering via mixtures of shifted asymmetric Laplace distributions , 2013 .

[53]  M. Forina,et al.  Multivariate data analysis as a discriminating method of the origin of wines , 2015 .

[54]  R. Sundberg.  Maximum Likelihood Theory for Incomplete Data from an Exponential Family , 2016 .

[55]  B. Everitt,et al.  Finite Mixture Distributions , 1981 .

[56]  Harry Joe,et al.  Generation of Random Clusters with Specified Degree of Separation , 2006, J. Classif..

[57]  Ryan P. Browne,et al.  Mixtures of Shifted AsymmetricLaplace Distributions , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Geoffrey J. McLachlan,et al.  On mixtures of skew normal and skew $$t$$-distributions , 2012, Adv. Data Anal. Classif..

[59]  Catherine B. Hurley,et al.  Clustering Visualizations of Multidimensional Data , 2004 .

[60]  Geoffrey J. McLachlan,et al.  Robust Cluster Analysis via Mixtures of Multivariate t-Distributions , 1998, SSPR/SPR.

[61]  Camille Roth,et al.  Natural Scales in Geographical Patterns , 2017, Scientific Reports.

[62]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[63]  Ryan P. Browne,et al.  Mixtures of Shifted Asymmetric Laplace Distributions , 2012 .

[64]  O. Barndorff-Nielsen,et al.  Infinite divisibility of the hyperbolic and generalized inverse Gaussian distributions , 1977 .

[65]  Christine M. Anderson-Cook,et al.  Book review: quantitative risk management: concepts, techniques and tools, revised edition, by A.F. McNeil, R. Frey and P. Embrechts. Princeton University Press, 2015, ISBN 978-0-691-16627-8, xix + 700 pp. , 2017, Extremes.

[66]  Ryan P. Browne,et al.  Model-Based Learning Using a Mixture of Mixtures of Gaussian and Uniform Distributions , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  William M. Rand,et al.  Objective Criteria for the Evaluation of Clustering Methods , 1971 .

[68]  Charles Bouveyron,et al.  Simultaneous model-based clustering and visualization in the Fisher discriminative subspace , 2011, Statistics and Computing.

[69]  A. Raftery,et al.  Variable Selection for Model-Based Clustering , 2006 .

[70]  Bernhard N Flury.  Multivariate Statistics: A Practical Approach , 1988 .

[71]  Dimitris Karlis,et al.  Model-based clustering with non-elliptically contoured distributions , 2009, Stat. Comput..

[72]  G. Mazo,et al.  Model-Based Clustering and Classification 4 , 2018 .

[73]  Paul D. McNicholas,et al.  Clustering with the multivariate normal inverse Gaussian distribution , 2016, Comput. Stat. Data Anal..

[74]  Ryan P. Browne,et al.  Mixtures of skew-t factor analyzers , 2013, Comput. Stat. Data Anal..

[75]  Luca Scrucca,et al.  Dimension reduction for model-based clustering , 2015, Stat. Comput..

[76]  G. Celeux,et al.  Regularized Gaussian Discriminant Analysis through Eigenvalue Decomposition , 1996 .

[77]  Geoffrey J. McLachlan,et al.  A mixture model-based approach to the clustering of microarray expression data , 2002, Bioinform..

[78]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[79]  M. Woodbury.  A missing information principle: theory and applications , 1972 .

[80]  G. J. McLachlan,et al.  9 The classification and mixture maximum likelihood approaches to cluster analysis , 1982, Classification, Pattern Recognition and Reduction of Dimensionality.

[81]  Paul D. McNicholas,et al.  Parsimonious skew mixture models for model-based clustering and classification , 2013, Comput. Stat. Data Anal..

[82]  B. Lindsay,et al.  The distribution of the likelihood ratio for mixtures of densities from the one-parameter exponential family , 1994 .

[83]  Peter Filzmoser,et al.  An Object-Oriented Framework for Robust Multivariate Analysis , 2009 .

[84]  William Nick Street,et al.  Breast Cancer Diagnosis and Prognosis Via Linear Programming , 1995, Oper. Res..

[85]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .