Testing for Unobserved Heterogeneity via K-Means Clustering

Clustering methods such as k-means have found widespread use in a variety of applications. This paper proposes a formal testing procedure to determine whether a null hypothesis of a single cluster, indicating homogeneity of the data, can be rejected in favor of multiple clusters. The test is simple to implement, valid under relatively mild conditions (including non-normality, and heterogeneity of the data in aspects beyond those in the clustering analysis), and applicable in a range of contexts (including clustering when the time series dimension is small, or clustering on parameters other than the mean). We verify that the test has good size control in finite samples, and we illustrate the test in applications to clustering vehicle manufacturers and U.S. mutual funds.

[1]  Ranjan Maitra,et al.  Bootstrapping for Significance of Compact Clusters in Multidimensional Datasets , 2012 .

[2]  M. Hashem Pesaran,et al.  Time Series and Panel Data Econometrics , 2015 .

[3]  Wei Fu,et al.  Estimating the Number of Clusters Using Cross-Validation , 2017, Journal of Computational and Graphical Statistics.

[4]  Junhui Wang Consistent selection of the number of clusters via crossvalidation , 2010 .

[5]  Elena Manresa,et al.  Grouped Patterns of Heterogeneity in Panel Data , 2015 .

[6]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[7]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[8]  William N. Goetzmann,et al.  Mutual Fund Styles , 1996 .

[9]  A. Gordaliza,et al.  Robustness Properties of k Means and Trimmed k Means , 1999 .

[10]  Michael T. Owyang,et al.  An Endogenously Clustered Factor Approach to International Business Cycles , 2012 .

[11]  George Karypis,et al.  A Comparison of Document Clustering Techniques , 2000 .

[12]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[13]  D. Botstein,et al.  Cluster analysis and display of genome-wide expression patterns. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Robert Tibshirani,et al.  Cluster Validation by Prediction Strength , 2005 .

[15]  A. Nobel,et al.  Statistical Significance of Clustering for High-Dimension, Low–Sample Size Data , 2008 .

[16]  D. Pollard A Central Limit Theorem for $k$-Means Clustering , 1982 .

[17]  Andrew J. Patton,et al.  What You See Is Not What You Get: The Costs of Trading Market Anomalies , 2019, Journal of Financial Economics.

[18]  Catherine A. Sugar,et al.  Finding the Number of Clusters in a Dataset , 2003 .

[19]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[20]  D. Pollard Strong Consistency of $K$-Means Clustering , 1981 .

[21]  C. Hansen Asymptotic properties of a robust variance matrix estimator for panel data when T is large , 2007 .

[22]  Sheridan Titman,et al.  On Persistence in Mutual Fund Performance , 1997 .

[23]  Siddheswar Ray,et al.  Determination of Number of Clusters in K-Means Clustering and Application in Colour Image Segmentation , 2000 .