A survey on context-aware mobile visual recognition

The phenomenal growth of the usage of mobile devices (e.g., mobile phones and tablet PCs) opens up a new service, namely mobile visual recognition, which has been widely used in many areas, such as mobile shopping and augmented reality. The rich contextual information (e.g., location, time and direction information), easily acquired by the mobile devices, provides useful clues to facilitate mobile visual recognition, including speeding up the recognition time and improving the recognition performance. This survey focuses on recent advances in Context-Aware Mobile Visual Recognition (CAMVR) and reviews related work regarding to different contextual information, recognition methods, recognition types, and various application scenarios. Finally, we discuss future research directions in this field.

[1]  Tao Chen,et al.  A multi-scale learning approach for landmark recognition using mobile devices , 2009, 2009 7th International Conference on Information, Communications and Signal Processing (ICICS).

[2]  Qi Tian,et al.  Multimedia search reranking: A literature survey , 2014, CSUR.

[3]  Tao Chen,et al.  Integrated Content and Context Analysis for Mobile Landmark Recognition , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Jiri Matas,et al.  Total recall II: Query expansion revisited , 2011, CVPR 2011.

[5]  Wen Gao,et al.  Towards low bit rate mobile visual search with multiple-channel coding , 2011, ACM Multimedia.

[6]  Edward Y. Chang,et al.  Extent: Inferring Image Metadata from Context and Content , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[7]  Andrew Zisserman,et al.  Three things everyone should know to improve object retrieval , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Tao Mei,et al.  Accurate sensing of scene geo-context via mobile visual localization , 2013, Multimedia Systems.

[9]  Bernd Girod,et al.  Streaming mobile augmented reality on mobile phones , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[10]  Lucas Paletta,et al.  Visual Object Detection for Mobile Road Sign Inventory , 2004, Mobile HCI.

[11]  Ke Gao,et al.  Geometric context-preserving progressive transmission in mobile visual search , 2012, ACM Multimedia.

[12]  Nina Runge,et al.  Keep an eye on your photos: automatic image tagging on mobile devices , 2014, MobileHCI '14.

[13]  Krista A. Ehinger,et al.  SUN database: Large-scale scene recognition from abbey to zoo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Yong-Hwan Lee,et al.  Photograph Indexing and Retrieval using Combined Geo-information and Visual Features , 2010, 2010 International Conference on Complex, Intelligent and Software Intensive Systems.

[15]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[16]  Chuan Qin,et al.  TagSense: Leveraging Smartphones for Automatic Image Tagging , 2014, IEEE Transactions on Mobile Computing.

[17]  Gordon Dodds,et al.  A PDA-Based System for Recognizing Buildings from User-Supplied Images , 2003, Mobile HCI Workshop on Mobile and Ubiquitous Information Access.

[18]  Rossana M. de Castro Andrade,et al.  Mobile Photo Recommendation and Logbook Generation Using Context-Tagged Images , 2014, IEEE MultiMedia.

[19]  Ying Wu,et al.  Mobile Product Image Search by Automatic Query Object Extraction , 2012, ECCV.

[20]  Hanqing Lu,et al.  Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Tao Chen,et al.  Context-Aware Discriminative Vocabulary Learning for Mobile Landmark Recognition , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  John K. Tsotsos,et al.  50 Years of object recognition: Directions forward , 2013, Comput. Vis. Image Underst..

[23]  C. V. Jawahar,et al.  Heritage app: annotating images on mobile phones , 2012, ICVGIP '12.

[24]  Luis Herranz,et al.  Semantic Features for Food Image Recognition with Geo-Constraints , 2014, 2014 IEEE International Conference on Data Mining Workshop.

[25]  Keiji Yanai,et al.  Real-Time Mobile Food Recognition System , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[26]  Wen-Huang Cheng,et al.  MobileQueue: an image-based queue card management system through augmented reality phones , 2012, UbiComp '12.

[27]  Junqing Yu,et al.  Efficient BOF Generation and Compression for On-Device Mobile Visual Location Recognition , 2014, IEEE MultiMedia.

[28]  Mor Naaman,et al.  From Where to What: Metadata Sharing for Digital Photographs with Geographic Coordinates , 2003, OTM.

[29]  Tao Chen,et al.  Content and context information fusion for mobile landmark recognition , 2011, 2011 8th International Conference on Information, Communications & Signal Processing.

[30]  Ning Zhang,et al.  Interactive mobile visual search for social activities completion using query image contextual model , 2012, 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP).

[31]  Changsheng Xu,et al.  Interaction Design for Mobile Visual Search , 2013, IEEE Transactions on Multimedia.

[32]  Zhen Li,et al.  Context-Aware Discriminative Vocabulary Learning for Mobile Landmark Recognition , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  Effrosini Kokiopoulou,et al.  Mobile Museum Guide Based on Fast SIFT Recognition , 2008, Adaptive Multimedia Retrieval.

[34]  A. Smeaton,et al.  Combination of content analysis and context features for digital photograph retrieval. , 2005 .

[35]  Jiebo Luo,et al.  Snap n' shop: Visual search-based mobile shopping made a breeze by machine and crowd intelligence , 2015, Proceedings of the 2015 IEEE 9th International Conference on Semantic Computing (IEEE ICSC 2015).

[36]  Tao Li,et al.  Direction-of-Arrival Estimation of Hydroacoustic Signals From Marine Vessels Containing Random and Sinusoidal Components , 2012, IEEE Signal Processing Letters.

[37]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[38]  Changsheng Xu,et al.  Mobile Landmark Search with 3D Models , 2014, IEEE Transactions on Multimedia.

[39]  Bernd Girod,et al.  Location coding for mobile image retrieval , 2009, MobiMedia.

[40]  Kate Saenko,et al.  Automatic mobile photo tagging using context , 2013, 2013 IEEE International Conference of IEEE Region 10 (TENCON 2013).

[41]  Shuang Wang,et al.  Geolocalized Modeling for Dish Recognition , 2015, IEEE Transactions on Multimedia.

[42]  Bernd Girod,et al.  Mobile product recognition , 2010, ACM Multimedia.

[43]  Rongrong Ji,et al.  Estimating viewing angles in mobile street view search , 2012, 2012 19th IEEE International Conference on Image Processing.

[44]  Lucas Paletta,et al.  Geo-indexed object recognition for mobile vision tasks , 2008, Mobile HCI.

[45]  Wen Gao,et al.  When codeword frequency meets geographical location , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[46]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[47]  Mor Naaman,et al.  Zonetag: Designing context-aware mobile media capture to increase participation , 2006 .

[48]  Bernd Girod,et al.  Tree Histogram Coding for Mobile Image Matching , 2009, 2009 Data Compression Conference.

[49]  Afshin Dehghan,et al.  Visual business recognition: a multimodal approach , 2013, MM '13.

[50]  Tao Chen,et al.  Context-aware vocabulary tree for mobile landmark recognition , 2015, J. Vis. Commun. Image Represent..

[51]  Yong-Hwan Lee,et al.  Mobile Image Retrieval Using Integration of Geo-sensing and Visual Descriptor , 2012, 2012 15th International Conference on Network-Based Information Systems.

[52]  Keiji Yanai,et al.  FoodCam-256: A Large-scale Real-time Mobile Food RecognitionSystem employing High-Dimensional Features and Compression of Classifier Weights , 2014, ACM Multimedia.

[53]  Alexander G. Hauptmann,et al.  Successful approaches in the TREC video retrieval evaluations , 2004, MULTIMEDIA '04.

[54]  Yongtian Wang,et al.  Mobile Visual Recognition on Smartphones , 2013, J. Sensors.

[55]  Keiji Yanai,et al.  Real-time mobile recipe recommendation system using food ingredient recognition , 2012, IMMPD '12.

[56]  Touradj Ebrahimi,et al.  Object-based tag propagation for semi-automatic annotation of images , 2010, MIR '10.

[57]  Min-Chun Hu,et al.  Learning and Recognition of On-Premise Signs From Weakly Labeled Street View Images , 2014, IEEE Transactions on Image Processing.

[58]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[59]  Tao Chen,et al.  Discriminative Soft Bag-of-Visual Phrase for Mobile Landmark Recognition , 2014, IEEE Transactions on Multimedia.

[60]  Mor Naaman,et al.  ZoneTag's Collaborative Tag Suggestions: What is This Person Doing in My Phone? , 2008, IEEE MultiMedia.

[61]  Zhen Li,et al.  Context-aware mobile image annotation for media search and sharing , 2013, Signal Process. Image Commun..

[62]  Qi Tian,et al.  Socio-mobile landmark recognition using local features with adaptive region selection , 2016, Neurocomputing.

[63]  Rongrong Ji,et al.  Learning from mobile contexts to minimize the mobile location search latency , 2013, Signal Process. Image Commun..

[64]  Jia Hao,et al.  Point of Interest Detection and Visual Distance Estimation for Sensor-Rich Video , 2014, IEEE Transactions on Multimedia.

[65]  Felix X. Yu.  Intelligent query formulation for mobile visual search , 2011, MM '11.

[66]  Bernd Girod,et al.  CHoG: Compressed histogram of gradients A low bit-rate feature descriptor , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Tao Mei,et al.  Robust and accurate mobile visual localization and its applications , 2013, TOMCCAP.

[68]  Weon-Geun Oh,et al.  Mobile Visual Search Applications , 2014 .

[69]  Tao Chen,et al.  Context-aware codebook learning for mobile landmark recognition , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[70]  Ming-Syan Chen,et al.  UbiShop: Commercial item recommendation using visual part-based object representation , 2016, Multimedia Tools and Applications.

[71]  Shih-Fu Chang,et al.  Mobile product search with Bag of Hash Bits and boundary reranking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[72]  Ming-Syan Chen,et al.  MOSRO: Enabling Mobile Sensing for Real-Scene Objects with Grid Based Structured Output Learning , 2014, MMM.

[73]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[74]  Nadjia Benblidia,et al.  Combining Context and Content for Automatic Image Annotation on Mobile Phones , 2013, 2013 International Conference on IT Convergence and Security (ICITCS).

[75]  Luis Herranz,et al.  A probabilistic model for food image recognition in restaurants , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[76]  Yiannis Kompatsiaris,et al.  A Comparative Study on Mobile Visual Recognition , 2013, MLDM.

[77]  Cordelia Schmid,et al.  TagProp: Discriminative metric learning in nearest neighbor models for image auto-annotation , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[78]  Jing Ren,et al.  Building a Large Scale Test Collection for Effective Benchmarking of Mobile Landmark Search , 2013, MMM.

[79]  Natasha Gelfand,et al.  Efficient Extraction of Robust Image Features on Mobile Devices , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[80]  Seungmin Rho,et al.  Location-Based Large-Scale Landmark Image Recognition Scheme for Mobile Devices , 2012, 2012 Third FTRA International Conference on Mobile, Ubiquitous, and Intelligent Computing.

[81]  Eckehard G. Steinbach,et al.  Exploiting prior knowledge in mobile visual location recognition , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[82]  Bernd Girod,et al.  Interframe Coding of Global Image Signatures for Mobile Augmented Reality , 2014, 2014 Data Compression Conference.

[83]  Wen Gao,et al.  Location Discriminative Vocabulary Coding for Mobile Landmark Search , 2011, International Journal of Computer Vision.

[84]  Huizhong Chen,et al.  The stanford mobile visual search data set , 2011, MMSys.

[85]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[86]  Ramesh C. Jain,et al.  Classification and annotation of digital photos using optical context data , 2008, CIVR '08.

[87]  Rongrong Ji,et al.  Active query sensing for mobile location search , 2011, ACM Multimedia.

[88]  Talmai Oliveira,et al.  A mobile, lightweight, poll-based food identification system , 2014, Pattern Recognit..

[89]  Robinson Piramuthu,et al.  Style Finder: Fine-Grained Clothing Style Detection and Retrieval , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[90]  Zhen Li,et al.  A Comparative Study of Mobile-Based Landmark Recognition Techniques , 2010, IEEE Intelligent Systems.

[91]  Mark S. Nixon,et al.  Mobile visual clothing search , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[92]  Junqing Yu,et al.  On-Device Mobile Visual Location Recognition by Integrating Vision and Inertial Sensors , 2013, IEEE Transactions on Multimedia.

[93]  Wen Gao,et al.  Learning Compact Visual Descriptor for Low Bit Rate Mobile Landmark Search , 2011, IJCAI.

[94]  Andrew Zisserman,et al.  Name that sculpture , 2012, ICMR.

[95]  Zhen Li,et al.  Content and Context Boosting for Mobile Landmark Recognition , 2012, IEEE Signal Processing Letters.

[96]  Kun Li,et al.  iScope: personalized multi-modality image search for mobile devices , 2009, MobiSys '09.

[97]  Tao Mei,et al.  Finding perfect rendezvous on the go: accurate mobile visual localization and its applications to routing , 2012, ACM Multimedia.

[98]  Johannes Schöning,et al.  iPiccer: automatically retrieving and inferring tagged location information from web repositories , 2009, Mobile HCI.

[99]  Hua Li,et al.  Mobile Search With Multimodal Queries , 2008, Proceedings of the IEEE.

[100]  Anas Al-Nuaimi,et al.  Mobile Visual Location Recognition , 2013 .

[101]  Xin Chen,et al.  City-scale landmark identification on mobile devices , 2011, CVPR 2011.

[102]  Joo-Hwee Lim,et al.  Scene Recognition with Camera Phones for Tourist Information Access , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[103]  Keiji Yanai,et al.  FoodCam: A Real-Time Mobile Food Recognition System Employing Fisher Vector , 2014, MMM.

[104]  Luc Van Gool,et al.  Object Recognition for the Internet of Things , 2008, IOT.

[105]  Tao Chen,et al.  Discriminative BoW Framework for Mobile Landmark Recognition , 2014, IEEE Transactions on Cybernetics.

[106]  Changsheng Xu,et al.  Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[107]  Lucas Paletta,et al.  A Mobile Vision System for Urban Detection with Informative Local Descriptors , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[108]  Wen-Huang Cheng,et al.  Augmenting mobile city-view image retrieval with context-rich user-contributed photos , 2011, ACM Multimedia.

[109]  Bernd Girod,et al.  Outdoors augmented reality on mobile phone using loxel-based visual feature organization , 2008, MIR '08.

[110]  Anne Verroust-Blondet,et al.  An android application for leaf-based plant identification , 2013, ICMR.

[111]  Shin'ichi Satoh,et al.  Annotation propagation in image databases using similarity graphs , 2013, ACM Trans. Multim. Comput. Commun. Appl..

[112]  Joo-Hwee Lim,et al.  Outdoor place recognition using compact local descriptors and multiple queries with user verification , 2007, ACM Multimedia.

[113]  Bernd Girod,et al.  Mobile Visual Search , 2011, IEEE Signal Processing Magazine.

[114]  Wen Gao,et al.  Towards Mobile Document Image Retrieval for Digital Library , 2014, IEEE Transactions on Multimedia.