Visual query processing for efficient image retrieval using a SOM-based filter-refinement scheme

Visual query processing is one of the new issues for content-based image retrieval. In this paper, we propose (i) a filter-refinement scheme based on a modified form of the self-organizing map and (ii) a new interactive approach for similarity matching in image retrieval based on visual query processing. Specifically, we first propose a new local membership function, which preserves the relationships between the input feature vectors of the images and their neighboring weight vectors, to project the high dimensional input feature vectors to a low dimensional grid. Then, all the input feature vectors are mapped and visualized in the 2D grid. The feature vector of the query image is mapped and visualized in the 2D grid as well. The users not only can visualize the locations of the query image and the image data in the database, but also visualize the locations of the relevant and irrelevant images. Next, the users retrieve the candidates from the 2D grid interactively through visual query processing in the filter phase. Finally, the query results are obtained from the candidates by performing similarity ranking in the original feature space during the refinement phase. In order to accelerate the query process, we use a hierarchical tree to index the weight vectors of the self-organizing map (SOM) units to reduce the computation cost for finding the best matching unit. Our experiments show that (i) the proposed approach works well on both synthetic datasets and image data, (ii) the proposed visual query processing approach is more efficient than conventional approaches and can enhance the overall interactive experience through fast feedback, and (iii) the filter-refinement scheme makes our proposed approach more robust than conventional approaches.

[1]  Cordelia Schmid,et al.  A maximum entropy framework for part-based texture and object recognition , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[2]  Ling Guan,et al.  Automatic machine interactions for content-based image retrieval using a self-organizing tree map architecture , 2002, IEEE Trans. Neural Networks.

[3]  Kimmo Kiviluoto,et al.  Topology preservation in self-organizing maps , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[4]  Sugata Ghosal,et al.  An image retrieval system with automatic query modification , 2002, IEEE Trans. Multim..

[5]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[6]  Elias Pampalk,et al.  Using Smoothed Data Histograms for Cluster Visualization in Self-Organizing Maps , 2002, ICANN.

[7]  Dacheng Tao,et al.  Bregman Divergence-Based Regularization for Transfer Subspace Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[8]  Hau-San Wong,et al.  Hierarchical indexing for 3D head model retrieval based on kernel PCA , 2005, Ninth International Conference on Information Visualisation (IV'05).

[9]  Qionghai Dai,et al.  Similarity-based online feature selection in content-based image retrieval , 2006, IEEE Transactions on Image Processing.

[10]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[11]  Rong Jin,et al.  A unified log-based relevance feedback scheme for image retrieval , 2006 .

[12]  Tae-Seong Kim,et al.  Content-based facial image retrieval using constrained independent component analysis , 2011, Inf. Sci..

[13]  Alex Pentland,et al.  Photobook: tools for content-based manipulation of image databases , 1994, Electronic Imaging.

[14]  G.G. Yen,et al.  Ranked centroid projection: a data visualization approach for self-organizing maps , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[15]  Hujun Yin,et al.  ViSOM - a novel method for multivariate data projection and structure visualization , 2002, IEEE Trans. Neural Networks.

[16]  Hyeran Byun,et al.  FRIP: a region-based image retrieval tool using automatic image segmentation and stepwise Boolean AND matching , 2005, IEEE Transactions on Multimedia.

[17]  Dacheng Tao,et al.  Evolutionary Cross-Domain Discriminative Hessian Eigenmaps , 2010, IEEE Transactions on Image Processing.

[18]  Rosalind W. Picard,et al.  Interactive Learning Using a "Society of Models" , 2017, CVPR 1996.

[19]  P. Navarrete,et al.  Interactive face retrieval using self-organizing maps , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[20]  Samuel Kaski,et al.  Bibliography of Self-Organizing Map (SOM) Papers: 1981-1997 , 1998 .

[21]  Yixin Chen,et al.  CLUE: cluster-based retrieval of images by unsupervised learning , 2005, IEEE Transactions on Image Processing.

[22]  James Ze Wang,et al.  SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Erkki Oja,et al.  PicSOM-self-organizing image retrieval with MPEG-7 content descriptors , 2002, IEEE Trans. Neural Networks.

[24]  B. S. Manjunath,et al.  NeTra: A toolbox for navigating large image databases , 1997, Multimedia Systems.

[25]  Tom Minka,et al.  Interactive learning with a "society of models" , 1997, Pattern Recognit..

[26]  A. Rauber,et al.  Investigation of alternative strategies and quality measures for controlling the growth process of the growing hierarchical self-organizing map , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[27]  Erkki Oja,et al.  Entropy-based measures for clustering and SOM topology preservation applied to content-based image indexing and retrieval , 2004, ICPR 2004.

[28]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Arnold W. M. Smeulders,et al.  Population-based incremental interactive concept learning for image retrieval by stochastic string segmentations , 2004, IEEE Transactions on Medical Imaging.

[30]  Konstantinos N. Plataniotis,et al.  Query feedback for interactive image retrieval , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Xindong Wu,et al.  Manifold elastic net: a unified framework for sparse dimension reduction , 2010, Data Mining and Knowledge Discovery.

[32]  James Ze Wang,et al.  Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Yanda Li,et al.  Self-organizing map as a new method for clustering and data analysis , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[34]  R. Rautkorpi,et al.  Content-based image retrieval of Web surface defects with PicSOM , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[35]  Weilin Huang,et al.  Adaptive nonlinear manifolds and their applications to pattern recognition , 2010, Inf. Sci..

[36]  Rosalind W. Picard,et al.  Finding similar patterns in large image databases , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[37]  Pasi Koikkalainen,et al.  Self-organizing hierarchical feature maps , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[38]  Dacheng Tao,et al.  Biased Discriminant Euclidean Embedding for Content-Based Image Retrieval , 2010, IEEE Transactions on Image Processing.

[39]  Andreas Rauber,et al.  The growing hierarchical self-organizing map: exploratory analysis of high-dimensional data , 2002, IEEE Trans. Neural Networks.

[40]  Tat-Seng Chua,et al.  NUS-WIDE: a real-world web image database from National University of Singapore , 2009, CIVR '09.

[41]  Xuelong Li,et al.  Geometric Mean for Subspace Selection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Shengli Xie,et al.  Coarse-to-Fine Boundary Location With a SOM-Like Method , 2010, IEEE Transactions on Neural Networks.

[43]  Nicolai Petkov,et al.  Comparison of texture features based on Gabor filters , 2002, IEEE Trans. Image Process..

[44]  King Ngi Ngan,et al.  A memory learning framework for effective image retrieval , 2005, IEEE Transactions on Image Processing.

[45]  Alan C. Bovik,et al.  Analysis of multichannel narrow-band filters for image texture segmentation , 1991, IEEE Trans. Signal Process..

[46]  Kyriakos Mouratidis,et al.  Conceptual partitioning: an efficient method for continuous nearest neighbor monitoring , 2005, SIGMOD '05.

[47]  Erkki Oja,et al.  Using MPEG-7 descriptors in image retrieval with self-organizing maps , 2002, Object recognition supported by user interaction for service robots.

[48]  Howard D. Wactlar,et al.  Informedia: improving access to digital video , 1994, INTR.

[49]  Juha Vesanto,et al.  SOM-based data visualization methods , 1999, Intell. Data Anal..

[50]  Meng Wang,et al.  MSRA-MM 2.0: A Large-Scale Web Multimedia Dataset , 2009, 2009 IEEE International Conference on Data Mining Workshops.

[51]  Xian-Sheng Hua,et al.  Active Reranking for Web Image Search , 2010, IEEE Transactions on Image Processing.

[52]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[53]  Andreas Rauber,et al.  Recent Advances with the Growing Hierarchical Self-Organizing Map , 2001, WSOM.

[54]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[55]  Esa Alhoniemi,et al.  Clustering of the self-organizing map , 2000, IEEE Trans. Neural Networks Learn. Syst..

[56]  Sung-Bae Cho,et al.  A human-oriented image retrieval system using interactive genetic algorithm , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[57]  Swarup Medasani,et al.  Content-based image retrieval based on a fuzzy approach , 2004, IEEE Transactions on Knowledge and Data Engineering.

[58]  Xuelong Li,et al.  Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Erkki Oja,et al.  Application of tree structured self-organizing maps in content-based image retrieval , 1999 .

[60]  Nam Chul Kim,et al.  Image retrieval using BDIP and BVLC moments , 2003, IEEE Trans. Circuits Syst. Video Technol..

[61]  Erkki Oja,et al.  PicSOM - content-based image retrieval with self-organizing maps , 2000, Pattern Recognit. Lett..

[62]  Cordelia Schmid,et al.  Semi-Local Affine Parts for Object Recognition , 2004, BMVC.

[63]  Alfred Ultsch,et al.  Data Mining and Knowledge Discovery with Emergent Self-Organizing Feature Maps for Multivariate Time Series , 1999 .

[64]  Xiaodong Gu,et al.  Color discrimination enhancement for dichromats using self-organizing color transformation , 2009, Inf. Sci..

[65]  Risto Miikkulainen,et al.  Incremental grid growing: encoding high-dimensional structure into a two-dimensional feature map , 1993, IEEE International Conference on Neural Networks.