Multi-graph multi-instance learning for object-based image and video retrieval

Object-based image retrieval has been an active research topic in recent years, in which a user is only interested in some object in the images. As one promising approach, graph-based multi-instance learning has attracted many researchers. The existing methods often conduct learning on one graph, either in image level or in region level. While in this paper, by considering both image- and region-level information at the same time, a novel method based on multi-graph multi-instance learning is proposed. Two graphs are constructed in our method, and the relationship between each image and its segmented regions is introduced into an optimization framework. Moreover, our method is further extended to video retrieval. By exploring the relationships between video shots, representative images, and segmented regions, it can deal with the case when training labels are only assigned in shot level. Experimental results on the SIVAL image benchmark and the TRECVID video set demonstrate the effectiveness of our proposal.

[1]  Wei-Ying Ma,et al.  Graph based multi-modality learning , 2005, ACM Multimedia.

[2]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[3]  Meng Wang,et al.  Optimizing multi-graph learning: towards a unified video annotation scheme , 2007, ACM Multimedia.

[4]  Jing Hua,et al.  Region-based Image Annotation using Asymmetrical Support Vector Machine-based Multiple-Instance Learning , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[5]  Wen Gao,et al.  Effective and efficient object-based image retrieval using visual phrases , 2006, MM '06.

[6]  Tao Mei,et al.  Multi-Layer Multi-Instance Learning for Video Concept Detection , 2008, IEEE Transactions on Multimedia.

[7]  Tat-Seng Chua,et al.  Image Annotation by Graph-Based Inference With Integrated Multiple/Single Instance Representations , 2010, IEEE Transactions on Multimedia.

[8]  Sally A. Goldman,et al.  MISSL: multiple-instance semi-supervised learning , 2006, ICML.

[9]  Derek Hoiem,et al.  Object-based image retrieval using the statistical structure of images , 2004, CVPR 2004.

[10]  Nicu Sebe,et al.  Content-based multimedia information retrieval: State of the art and challenges , 2006, TOMCCAP.

[11]  Fei Wang,et al.  Interactive localized content based image retrieval with multiple-instance active learning , 2010, Pattern Recognit..

[12]  Changshui Zhang,et al.  Instance-level Semisupervised Multiple Instance Learning , 2008, AAAI.

[13]  Rujie Liu,et al.  Graph-based multiple-instance learning with instance weighting for image retrieval , 2011, 2011 18th IEEE International Conference on Image Processing.

[14]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Qi Tian,et al.  Semantic retrieval of video - review of research on video retrieval in meetings, movies and broadcast news, and sports , 2006, IEEE Signal Processing Magazine.

[16]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[17]  Yixin Chen,et al.  Image Categorization by Learning and Reasoning with Regions , 2004, J. Mach. Learn. Res..

[18]  Thomas G. Dietterich,et al.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles , 1997, Artif. Intell..

[19]  Pietro Perona,et al.  A Bayesian hierarchical model for learning natural scene categories , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[20]  Jingrui He,et al.  Manifold-ranking based image retrieval , 2004, MULTIMEDIA '04.

[21]  Zhi-Hua Zhou,et al.  On the relation between multi-instance learning and semi-supervised learning , 2007, ICML '07.

[22]  Yixin Chen,et al.  MILES: Multiple-Instance Learning via Embedded Instance Selection , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Wu-Jun Li,et al.  Localized content-based image retrieval through evidence region identification , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Shuenn-Ren Cheng,et al.  Multiple-instance content-based image retrieval employing isometric embedded similarity measure , 2009, Pattern Recognit..

[25]  Dan Zhang,et al.  Localized Content-Based Image Retrieval Using Semi-Supervised Multiple Instance Learning , 2007, ACCV.

[26]  Nenghai Yu,et al.  Multiple-instance ranking: Learning to rank images for image retrieval , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Xian-Sheng Hua,et al.  Typicality ranking via semi-supervised multiple-instance learning , 2007, ACM Multimedia.

[28]  Yihong Gong,et al.  Locality-constrained Linear Coding for image classification , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[29]  Wessel Kraaij,et al.  TRECVID-2009 high-level feature task: Overview (slides0 , 2005 .

[30]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[31]  Thomas Hofmann,et al.  Kernel Methods for Missing Variables , 2005, AISTATS.

[32]  Yangqing Jia,et al.  Instance-level Semisupervised Multiple Instance Learning , 2008, AAAI.

[33]  Pietro Perona,et al.  Learning object categories from Google's image search , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[34]  Meng Wang,et al.  Video annotation by graph-based learning with neighborhood similarity , 2007, ACM Multimedia.

[35]  Hui Zhang,et al.  Localized Content-Based Image Retrieval , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[37]  De Xu,et al.  Transductive Multi-Instance Multi-Label learning algorithm with application to automatic image annotation , 2010, Expert Syst. Appl..