Using Multi-Modal Semantic Association Rules to fuse keywords and visual features automatically for Web image retrieval

A recent trend for image search is to fuse the two basic modalities of Web images, i.e., textual features (usually represented by keywords) and visual features for retrieval. The key issue is how to associate the two modalities for fusion. In this paper, a new approach based on Multi-Modal Semantic Association Rule (MMSAR) is proposed to fuse keywords and visual features automatically for Web image retrieval. A MMSAR contains a single keyword and several visual feature clusters, which crosses and associates the two modalities of Web images. A customized frequent itemsets mining algorithm is designed for the particular MMSARs based on the existing inverted file, and a new support-confidence framework is defined for the mining algorithm. Based on the mined MMSARs, the keywords and the visual features are fused automatically in the retrieval process. The proposed approach not only remarkably improves the retrieval precision, but also has fast response time. The experiments are carried out in a Web image retrieval system, VAST (VisuAl & SemanTic image search), and the results show the superiority and effectiveness of the proposed approach.

[1]  James Ze Wang,et al.  SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Richa Singh,et al.  Robust memory-efficient data level information fusion of multi-modal biometric images , 2007, Inf. Fusion.

[3]  Heung-Kyu Lee,et al.  Majority Based Ranking Approach in Web Image Retrieval , 2003, CIVR.

[4]  Rong Jin,et al.  A unified log-based relevance feedback scheme for image retrieval , 2006 .

[5]  Gerald Salton,et al.  Automatic text processing , 1988 .

[6]  Jorma Laaksonen,et al.  Using Long-Term Learning to Improve Efficiency of Content-Based Image Retrieval , 2003, PRIS.

[7]  C.-C. Jay Kuo,et al.  Texture analysis and classification with tree-structured wavelet transform , 1993, IEEE Trans. Image Process..

[8]  Steven J. Phillips.  Acceleration of K-Means and Related Clustering Algorithms , 2002, ALENEX.

[9]  Sharad Mehrotra,et al.  WebMARS: a multimedia search engine , 1999, Electronic Imaging.

[10]  Hai Jin,et al.  A Flexible and Extensible Framework for Web Image Retrieval System , 2006, Advanced Int'l Conference on Telecommunications and Int'l Conference on Internet and Web Applications and Services (AICT-ICIW'06).

[11]  Pratim Ghosh,et al.  CORTINA: Searching a 10 Million + Images Database , 2007 .

[12]  Peiling Wang,et al.  Mining longitudinal web queries: Trends and patterns , 2003, J. Assoc. Inf. Sci. Technol..

[13]  Katsumi Tanaka,et al.  Retrieving Web images based on their usage context for augmenting ubiquitous contents , 2003, 2003 IEEE Pacific Rim Conference on Communications Computers and Signal Processing (PACRIM 2003) (Cat. No.03CH37490).

[14]  Michael J. Swain,et al.  WebSeer: An Image Search Engine for the World Wide Web , 1996 .

[15]  Rong Jin,et al.  Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval , 2005, 21st International Conference on Data Engineering Workshops (ICDEW'05).

[16]  William I. Grosky,et al.  Narrowing the semantic gap - improved text-based web document retrieval using visual features , 2002, IEEE Trans. Multim..

[17]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Mingjing Li,et al.  Web mining for web image retrieval , 2001 .

[19]  Arnold W. M. Smeulders,et al.  PicToSeek: combining color and shape invariant features for image retrieval , 2000, IEEE Trans. Image Process..

[20]  Belur V. Dasarathy.  A special issue on image fusion: Advances in the state of the art , 2007, Inf. Fusion.

[21]  Ramakrishnan Srikant,et al.  Fast Algorithms for Mining Association Rules in Large Databases , 1994, VLDB.

[22]  Jitendra Malik,et al.  Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Thijs Westerveld,et al.  Image Retrieval: Content versus Context , 2000, RIAO.

[24]  Robert Villa,et al.  The effectiveness of query-specific hierarchic clustering in information retrieval , 2002, Inf. Process. Manag..

[25]  Luo Si,et al.  Effect of varying number of documents in blind feedback: analysis of the 2003 NRRC RIA workshop "bf_numdocs" experiment suite , 2004, SIGIR '04.

[26]  Monika Henzinger,et al.  Analysis of a very large web search engine query log , 1999, SIGF.

[27]  Stavri G. Nikolov,et al.  Image fusion: Advances in the state of the art , 2007, Inf. Fusion.

[28]  Heung-Kyu Lee,et al.  Web image retrieval using majority-based ranking approach , 2006, Multimedia Tools and Applications.

[29]  Jingrui He,et al.  Pseudo Relevance Feedback Based on Iterative Probabilistic One-Class SVMs in Web Image Retrieval , 2004, PCM.

[30]  Heung-Kyu Lee,et al.  A Ranking Algorithm Using Dynamic Clustering for Content-Based Image Retrieval , 2002, CIVR.

[31]  Aggelos K. Katsaggelos,et al.  Hybrid image segmentation using watersheds and fast region merging , 1998, IEEE Trans. Image Process..

[32]  Wei-Ying Ma,et al.  Improving pseudo-relevance feedback in web information retrieval using web page segmentation , 2003, WWW '03.

[33]  S. Sclaroff,et al.  Combining textual and visual cues for content-based image retrieval on the World Wide Web , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[34]  Bo Zhang,et al.  A unified framework for image retrieval using keyword and visual features , 2005, IEEE Transactions on Image Processing.

[35]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[36]  Wei-Ying Ma,et al.  Learning a semantic space from user's relevance feedback for image retrieval , 2003, IEEE Trans. Circuits Syst. Video Technol..

[37]  Rong Yan,et al.  Multimedia Search with Pseudo-relevance Feedback , 2003, CIVR.

[38]  Leonidas J. Guibas,et al.  A metric for distributions with applications to image databases , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[39]  Christos Faloutsos,et al.  QBIC project: querying images by content, using color, texture, and shape , 1993, Electronic Imaging.

[40]  Ferenc Bodon,et al.  A fast APRIORI implementation , 2003, FIMI.

[41]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[42]  Djemel Ziou,et al.  Image Retrieval from the World Wide Web: Issues, Techniques, and Systems , 2004, CSUR.