Mobile multi-view object image search

High user interaction capability of mobile devices can help improve the accuracy of mobile visual search systems. At query time, it is possible to capture multiple views of an object from different viewing angles and at different scales with the mobile device camera to obtain richer information about the object compared to a single view and hence return more accurate results. Motivated by this, we propose a new multi-view visual query model on multi-view object image databases for mobile visual search. Multi-view images of objects acquired by the mobile clients are processed and local features are sent to a server, which combines the query image representations with early/late fusion methods and returns the query results. We performed a comprehensive analysis of early and late fusion approaches using various similarity functions, on an existing single view and a new multi-view object image database. The experimental results show that multi-view search provides significantly better retrieval accuracy compared to traditional single view search.

[1]  Junqing Yu,et al.  Efficient BOF Generation and Compression for On-Device Mobile Visual Location Recognition , 2014, IEEE MultiMedia.

[2]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[3]  Yan-Ying Chen,et al.  Enabling low bitrate mobile visual recognition: a performance versus bandwidth evaluation , 2013, MM '13.

[4]  Dawei Li,et al.  EMOD: an efficient on-device mobile visual search system , 2015, MMSys.

[5]  Andrew Zisserman,et al.  Multiple queries for large scale specific object retrieval , 2012, BMVC.

[6]  Masoud Mazloom,et al.  Querying for video events by semantic signatures from few examples , 2013, MM '13.

[7]  Lei Zhu,et al.  Supporting multi-example image queries in image databases , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[8]  Scott T. Acton,et al.  An image retrieval algorithm using multiple query images , 2003, Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings..

[9]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Sung-Hyuk Cha.  Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[11]  徐梦溪,et al.  Network video monitoring system based on OpenCV (open source computer vision library) , 2011 .

[12]  Ying Wu,et al.  Mobile Product Image Search by Automatic Query Object Extraction , 2012, ECCV.

[13]  Chengcui Zhang,et al.  An Online Multiple Instance Learning System for Semantic Image Retrieval , 2007, Ninth IEEE International Symposium on Multimedia Workshops (ISMW 2007).

[14]  Xueming Qian,et al.  Mobile image retrieval using multi-photos as query , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[15]  Victor S. Lempitsky,et al.  Aggregating Deep Convolutional Features for Image Retrieval , 2015, ArXiv.

[16]  Bernard Mérialdo,et al.  Fusion methods for multi-modal indexing of web data , 2013, 2013 14th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS).

[17]  Rongrong Ji,et al.  Active query sensing for mobile location search , 2011, ACM Multimedia.

[18]  G. Griffin,et al.  Caltech-256 Object Category Dataset , 2007 .

[19]  Hai Jin,et al.  Landmark Classification With Hierarchical Multi-Modal Exemplar Feature , 2015, IEEE Transactions on Multimedia.

[20]  Cordelia Schmid,et al.  Local Convolutional Features with Unsupervised Training for Image Retrieval , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Chu-Hui Lee,et al.  A Multi-query Strategy forContent-based Image Retrieval , 2011 .

[22]  Ming Yang,et al.  Query Specific Fusion for Image Retrieval , 2012, ECCV.

[23]  Guy Shani,et al.  A Survey of Accuracy Evaluation Metrics of Recommendation Tasks , 2009, J. Mach. Learn. Res..

[24]  Yang Wang,et al.  JIGSAW: interactive mobile visual search with multimodal queries , 2011, ACM Multimedia.

[25]  Rongrong Ji,et al.  Active query sensing: Suggesting the best query view for mobile visual search , 2012, TOMCCAP.

[26]  Bernd Girod,et al.  Mobile Visual Search , 2011, IEEE Signal Processing Magazine.

[27]  Christoph H. Lampert.  Detecting objects in large image collections and videos by efficient subimage retrieval , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[28]  Kannan Balakrishnan,et al.  Multi-Query Content Based Image Retrieval System using Local Binary Patterns , 2011 .

[29]  Henning Biermann,et al.  Regions-of-Interest and Spatial Layout for Content-Based Image Retrieval , 2001, Multimedia Tools and Applications.

[30]  Ning Zhang,et al.  TapTell: Interactive visual search for mobile task recommendation , 2015, J. Vis. Commun. Image Represent..

[31]  Bernd Girod,et al.  Mobile Visual Search: Architectures, Technologies, and the Emerging MPEG Standard , 2011, IEEE MultiMedia.

[32]  Qi Tian,et al.  Towards Codebook-Free: Scalable Cascaded Hashing for Mobile Image Search , 2014, IEEE Transactions on Multimedia.

[33]  Xin Chen,et al.  City-scale landmark identification on mobile devices , 2011, CVPR 2011.

[34]  Changsheng Xu,et al.  Mobile Landmark Search with 3D Models , 2014, IEEE Transactions on Multimedia.

[35]  Bernd Girod,et al.  Memory-Efficient Image Databases for Mobile Visual Search , 2014, IEEE MultiMedia.