Unsupervised Learning and Clustering

Unsupervised learning is very important in the processing of multimedia content as clustering or partitioning of data in the absence of class labels is often a requirement. This chapter begins with a review of the classic clustering techniques of k-means clustering and hierarchical clustering. Modern advances in clustering are covered with an analysis of kernel-based clustering and spectral clustering. One of the most popular unsupervised learning techniques for processing multimedia content is the self-organizing map, so a review of self-organizing maps and variants is presented in this chapter. The absence of class labels in unsupervised learning makes the question of evaluation and cluster quality assessment more complicated than in supervised learning. So this chapter also includes a comprehensive analysis of cluster validity assessment techniques.

[1]  Brian W. Kernighan,et al.  An efficient heuristic procedure for partitioning graphs , 1970, Bell Syst. Tech. J..

[2]  Joachim M. Buhmann,et al.  A Resampling Approach to Cluster Validation , 2002, COMPSTAT.

[3]  Risto Mukkulainen,et al.  Script Recognition with Hierarchical Feature Maps , 1990 .

[4]  Erkki Oja,et al.  Engineering applications of the self-organizing map , 1996, Proc. IEEE.

[5]  David G. Stork,et al.  Pattern Classification , 1973 .

[6]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Inderjit S. Dhillon,et al.  Co-clustering documents and words using bipartite spectral graph partitioning , 2001, KDD '01.

[8]  Isabelle Guyon,et al.  A Stability Based Method for Discovering Structure in Clustered Data , 2001, Pacific Symposium on Biocomputing.

[9]  T. Kohonen,et al.  Bibliography of Self-Organizing Map SOM) Papers: 1998-2001 Addendum , 2003 .

[10]  H. B. Barlow,et al.  Unsupervised Learning , 1989, Neural Computation.

[11]  Sudipto Guha,et al.  CURE: an efficient clustering algorithm for large databases , 1998, SIGMOD '98.

[12]  George Karypis,et al.  Evaluation of hierarchical clustering algorithms for document datasets , 2002, CIKM '02.

[13]  E. Forgy,et al.  Cluster analysis of multivariate data : efficiency versus interpretability of classifications , 1965 .

[14]  Susan T. Dumais,et al.  Using Linear Algebra for Intelligent Information Retrieval , 1995, SIAM Rev..

[15]  William M. Rand,et al.  Objective Criteria for the Evaluation of Clustering Methods , 1971 .

[16]  Eytan Domany,et al.  Resampling Method for Unsupervised Estimation of Cluster Validity , 2001, Neural Computation.

[17]  A. Hoffman,et al.  Lower bounds for the partitioning of graphs , 1973 .

[18]  John W. Sammon,et al.  A Nonlinear Mapping for Data Structure Analysis , 1969, IEEE Transactions on Computers.

[19]  Jeng-Shyang Pan,et al.  Improved partial distance search for k nearest-neighbor classification , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[20]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[21]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[22]  P. Rousseeuw Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[23]  Kenneth M. Hall An r-Dimensional Quadratic Placement Algorithm , 1970 .

[24]  P. Jaccard THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1 , 1912 .

[25]  Joachim M. Buhmann,et al.  Stability-Based Validation of Clustering Solutions , 2004, Neural Computation.

[26]  Joseph T. Chang,et al.  Spectral biclustering of microarray data: coclustering genes and conditions. , 2003, Genome research.

[27]  J. Dunn Well-Separated Clusters and Optimal Fuzzy Partitions , 1974 .

[28]  Ioan Tabus,et al.  Cluster Structure Inference Based on Clustering Stability with Applications to Microarray Data Analysis , 2004, EURASIP J. Adv. Signal Process..

[29]  S. Dudoit,et al.  A prediction-based resampling method for estimating the number of clusters in a dataset , 2002, Genome Biology.

[30]  Francisco Azuaje,et al.  Cluster validation techniques for genome expression data , 2003, Signal Process..

[31]  Risto Miikkulainen,et al.  Incremental grid growing: encoding high-dimensional structure into a two-dimensional feature map , 1993, IEEE International Conference on Neural Networks.

[32]  Risto Miikkulainen,et al.  Script Recognition with Hierarchical Feature Maps , 1992 .

[33]  Hanan Samet,et al.  K-Nearest Neighbor Finding Using MaxNearestDist , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Inderjit S. Dhillon,et al.  Kernel k-means: spectral clustering and normalized cuts , 2004, KDD.

[35]  Tomi Kinnunen,et al.  Improving K-Means by Outlier Removal , 2005, SCIA.

[36]  Robert Tibshirani,et al.  Cluster Validation by Prediction Strength , 2005 .

[37]  Kun Huang,et al.  A unifying theorem for spectral embedding and clustering , 2003, AISTATS.

[38]  Jianhong Wu,et al.  Data clustering - theory, algorithms, and applications , 2007 .

[39]  J. C. Dunn,et al.  A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters , 1973 .

[40]  L. Hubert,et al.  A general statistical framework for assessing categorical clustering in free recall. , 1976 .

[41]  Peter J. Rousseeuw,et al.  Finding Groups in Data: An Introduction to Cluster Analysis , 1990 .

[42]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[43]  RICHARD C. DUBES,et al.  How many clusters are best? - An experiment , 1987, Pattern Recognit..

[44]  T. Caliński,et al.  A dendrite method for cluster analysis , 1974 .

[45]  Chinatsu Aone,et al.  Fast and effective text mining using linear-time document clustering , 1999, KDD '99.

[46]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[47]  C. Mallows,et al.  A Method for Comparing Two Hierarchical Clusterings , 1983 .

[48]  M. Fiedler Algebraic connectivity of graphs , 1973 .

[49]  Alex Pothen,et al.  PARTITIONING SPARSE MATRICES WITH EIGENVECTORS OF GRAPHS* , 1990 .

[50]  Joachim M. Buhmann,et al.  Path-Based Clustering for Grouping of Smooth Curves and Texture Segmentation , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[52]  Chris H. Q. Ding,et al.  Cluster merging and splitting in hierarchical clustering algorithms , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[53]  James C. Bezdek,et al.  Cluster validation with generalized Dunn's indices , 1995, Proceedings 1995 Second New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems.

[54]  Bernd Fritzke Growing Grid — a self-organizing network with constant neighborhood range and adaptation strength , 1995, Neural Processing Letters.

[55]  Andreas Rauber,et al.  The growing hierarchical self-organizing map: exploratory analysis of high-dimensional data , 2002, IEEE Trans. Neural Networks.

[56]  Jianbo Shi,et al.  Multiclass spectral clustering , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[57]  George Karypis,et al.  A Comparison of Document Clustering Techniques , 2000 .

[58]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[59]  Tian Zhang,et al.  BIRCH: an efficient data clustering method for very large databases , 1996, SIGMOD '96.

[60]  M. Aizerman,et al.  Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning , 1964 .

[61]  Samuel Kaski,et al.  Bibliography of Self-Organizing Map (SOM) Papers: 1981-1997 , 1998 .

[62]  H. Kuhn The Hungarian method for the assignment problem , 1955 .

[63]  Dieter Merkl,et al.  Exploration of text collections with hierarchical feature maps , 1997, SIGIR '97.

[64]  Bernd Fritzke Growing Cell Structures – a Self-organizing Network in k Dimensions , 1992 .

[65]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[66]  G. W. Milligan,et al.  An examination of procedures for determining the number of clusters in a data set , 1985 .

[67]  A Gordon,et al.  Classification, 2nd Edition , 1999 .

[68]  Andreas Rauber,et al.  The SOMLib Digital Library System , 1999, ECDL.

[69]  Donald W. Bouldin,et al.  A Cluster Separation Measure , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.