Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying

Retrieving images from large and varied collections using image content as a key is a challenging and important problem. We present a new image representation that provides a transformation from the raw pixel data to a small set of image regions that are coherent in color and texture. This "Blobworld" representation is created by clustering pixels in a joint color-texture-position feature space. The segmentation algorithm is fully automatic and has been run on a collection of 10,000 natural images. We describe a system that uses the Blobworld representation to retrieve images from this collection. An important aspect of the system is that the user is allowed to view the internal representation of the submitted image and the query results. Similar systems do not offer the user this view into the workings of the system; consequently, query results from these systems can be inexplicable, despite the availability of knobs for adjusting the similarity metrics. By finding image regions that roughly correspond to objects, we allow querying at the level of objects rather than global image properties. We present results indicating that querying for images using Blobworld produces higher precision than does querying using color and texture histograms of the entire image in cases where the image contains distinctive objects.

[1]  M. Wertheimer.  Laws of organization in perceptual forms. , 1938 .

[2]  Gunther Wyszecki,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulae, 2nd Edition , 2000 .

[3]  J. Cohen,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulas , 1968 .

[4]  Larry D. Hostetler,et al.  The estimation of the gradient of a density function, with applications in pattern recognition , 1975, IEEE Trans. Inf. Theory.

[5]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[6]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[7]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[8]  G. Wyszecki,et al.  Color Science Concepts and Methods , 1982 .

[9]  Anil K. Jain,et al.  Unsupervised texture segmentation using Gabor filters , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[10]  P Perona,et al.  Preattentive texture discrimination with early vision mechanisms. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[11]  Johan Wiklund,et al.  Multidimensional Orientation Estimation with Applications to Texture Analysis and Optical Flow , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Anil K. Jain,et al.  Unsupervised texture segmentation using Gabor filters , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[14]  Donna K. Harman,et al.  Relevance Feedback and Other Query Modification Techniques , 1992, Information retrieval (Boston).

[15]  John P. Oakley,et al.  Storage and Retrieval for Image and Video Databases , 1993 .

[16]  Wolfgang Förstner,et al.  A Framework for Low Level Feature Extraction , 1994, ECCV.

[17]  Hans Knutsson,et al.  Signal processing for computer vision , 1994 .

[18]  Michael J. Swain,et al.  The Capacity and the Sensitivity of Color Histogram Indexing , 1994 .

[19]  Martial Hebert,et al.  Object Representation in Computer Vision , 1994, Lecture Notes in Computer Science.

[20]  Don R. Hush,et al.  Query by image example: The CANDID approach , 1995 .

[21]  Shih-Fu Chang,et al.  Single color extraction and image query , 1995, Proceedings., International Conference on Image Processing.

[22]  David Salesin,et al.  Fast multiresolution image querying , 1995, SIGGRAPH.

[23]  T. M. Cannon,et al.  Query by image example: the comparison algorithm for navigating digital image databases (CANDID) approach , 1995, Electronic imaging.

[24]  Glenn Healey,et al.  Markov Random Field Models for Unsupervised Segmentation of Textured Color Images , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  J. Ashley,et al.  Automatic and Semi-Automatic Methods for Image Annotation and Retrieval in QBIC , 1995 .

[26]  Michael Stonebraker,et al.  Chabot: Retrieval from a Relational Database of Images , 1995, Computer.

[27]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[28]  James Lee Hafner,et al.  Efficient Color Histogram Indexing for Quadratic Form Distance Functions , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Harpreet S. Sawhney,et al.  Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding , 1995, Proceedings of IEEE International Conference on Computer Vision.

[30]  Dragutin Petkovic,et al.  Automatic and semiautomatic methods for image annotation and retrieval in query by image content (QBIC) , 1995, Electronic Imaging.

[31]  Shih-Fu Chang,et al.  Tools and techniques for color image retrieval , 1996, Electronic Imaging.

[32]  W. Eric L. Grimson,et al.  Adaptive Segmentation of MRI Data , 1995, CVRMed.

[33]  Jitendra Malik,et al.  Detecting, localizing and grouping repeated scene elements from an image , 1996, ECCV.

[34]  Cordelia Schmid,et al.  Combining greyvalue invariants with local constraints for object recognition , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Edward H. Adelson,et al.  A unified mixture framework for motion segmentation: incorporating spatial coherence and estimating the number of models , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Martial Hebert,et al.  Object Representation in Computer Vision II , 1996, Lecture Notes in Computer Science.

[37]  Rosalind W. Picard,et al.  Interactive Learning Using a "Society of Models" , 2017, CVPR 1996.

[38]  David A. Forsyth,et al.  Body plans , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[39]  D. Forsyth,et al.  Searching for Digital Pictures , 1997 .

[40]  Amarnath Gupta,et al.  Visual information retrieval , 1997, CACM.

[41]  Tom Minka,et al.  Interactive learning with a "society of models" , 1997, Pattern Recognit..

[42]  Serge J. Belongie,et al.  Region-based image querying , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[43]  Jing Huang,et al.  Image indexing using color correlograms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[44]  W. Eric L. Grimson,et al.  Configuration based scene classification and image indexing , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[45]  Markus A. Stricker,et al.  Spectral covariance and fuzzy regions for image indexing , 1997, Machine Vision and Applications.

[46]  Chuen-Tsai Sun,et al.  Neuro-fuzzy And Soft Computing: A Computational Approach To Learning And Machine Intelligence [Books in Brief] , 1997, IEEE Transactions on Neural Networks.

[47]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[48]  Joachim M. Buhmann,et al.  Multiscale annealing for real-time unsupervised texture segmentation , 1997, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[49]  Jitendra Malik,et al.  Color- and texture-based image segmentation using EM and its application to content-based image retrieval , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[50]  B. S. Manjunath,et al.  NeTra: A toolbox for navigating large image databases , 1997, Multimedia Systems.

[51]  Jitendra Malik,et al.  Blobworld: A System for Region-Based Image Indexing and Retrieval , 1999, VISUAL.

[52]  Carlo Tomasi,et al.  Edge, Junction, and Corner Detection Using Color Distributions , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[55]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[56]  Tony Lindeberg,et al.  Direct computation of shape cues using scale-adapted spatial derivative operators , 1996, International Journal of Computer Vision.