Image Location Inference by Multisaliency Enhancement

Locations of images have been widely used in many application scenarios for large geotagged image corpora. As to images that are not geographically tagged, we estimate their locations with the help of the large geotagged image set by content-based image retrieval. Bag-of-words image representation has been utilized widely. However, the individual visual word-based image retrieval approach is not effective in expressing the salient relationships of image region. In this paper, we present an image location estimation approach by multisaliency enhancement. We first extract region-of-interests (ROIs) by mean-shift clustering on the visual words and salient map of the image based on which we further determine the importance of the ROI. Then, we describe each ROI by the spatial descriptors of visual words. Finally, region-based visual phrases are generated to further enhance the saliency in image location estimation. Experiments show the effectiveness of our proposed approach.

[1]  Qi Tian,et al.  Spatial coding for large scale partial-duplicate web image search , 2010, ACM Multimedia.

[2]  Ling Shao,et al.  Feature Learning for Image Classification Via Multiobjective Genetic Programming , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[3]  K. K. More,et al.  Interactive Multimodal Visual Search on Mobile Device , 2015 .

[4]  Jianping Fan,et al.  Image collection summarization via dictionary learning for sparse representation , 2013, Pattern Recognit..

[5]  Xiaochun Cao,et al.  Cluster-Based Co-Saliency Detection , 2013, IEEE Transactions on Image Processing.

[6]  Jan-Michael Frahm,et al.  3D model search and pose estimation from single images using VIP features , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[7]  Steven Schockaert,et al.  Ghent University at the 2011 Placing Task , 2011, MediaEval.

[8]  Yuan Yan Tang,et al.  GPS Estimation from Users' Photos , 2013, MMM.

[9]  Xiaochun Cao,et al.  Co-Saliency Detection via Base Reconstruction , 2014, ACM Multimedia.

[10]  Tao Mei,et al.  Author Topic Model-Based Collaborative Filtering for Personalized POI Recommendations , 2015, IEEE Transactions on Multimedia.

[11]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[12]  Meng Wang,et al.  Enhancing Sketch-Based Image Retrieval by Re-Ranking and Relevance Feedback , 2016, IEEE Transactions on Image Processing.

[13]  Xin Li,et al.  Interactive object-based image retrieval and annotation on iPad , 2013, Multimedia Tools and Applications.

[14]  Alexei A. Efros,et al.  Image sequence geolocation with human travel priors , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[15]  Xueming Qian,et al.  Visual summarization of landmarks via viewpoint modeling , 2012, 2012 19th IEEE International Conference on Image Processing.

[16]  Yang Song,et al.  Tour the world: Building a web-scale landmark recognition engine , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  O. K. Gowrishankar,et al.  Personalized Travel Sequence Recommendation on Multi-Source Big Social Media , 2016, IEEE Transactions on Big Data.

[18]  Ling Shao,et al.  Cosaliency Detection Based on Intrasaliency Prior Transfer and Deep Intersaliency Mining , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Geert-Jan Houben,et al.  Placing images on the world map: a microblog-based enrichment approach , 2012, SIGIR '12.

[20]  Dieter Schmalstieg,et al.  Discriminative Feature-to-Point Matching in Image-Based Localization , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Luc Van Gool,et al.  World-scale mining of objects and events from community photo collections , 2008, CIVR '08.

[22]  Michael Isard,et al.  Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[23]  Tao Mei,et al.  Finding perfect rendezvous on the go: accurate mobile visual localization and its applications to routing , 2012, ACM Multimedia.

[24]  Arnold W. M. Smeulders,et al.  Visual synonyms for landmark image retrieval , 2012, Comput. Vis. Image Underst..

[25]  Yang Song,et al.  Tour the world: a technical demonstration of a web-scale landmark recognition engine , 2009, ACM Multimedia.

[26]  Chao Li,et al.  Co-saliency detection via looking deep and wide , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  King Ngi Ngan,et al.  A Co-Saliency Model of Image Pairs , 2011, IEEE Transactions on Image Processing.

[28]  Yuan Yan Tang,et al.  GPS Estimation for Places of Interest From Social Users' Uploaded Photos , 2013, IEEE Transactions on Multimedia.

[29]  Yuan Yan Tang,et al.  Landmark Summarization With Diverse Viewpoints , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[30]  Nicu Sebe,et al.  Feature Selection for Multimedia Analysis by Sharing Information Among Multiple Tasks , 2013, IEEE Transactions on Multimedia.

[31]  Xueming Qian,et al.  Image Location Estimation by Salient Region Matching , 2015, IEEE Transactions on Image Processing.

[32]  Xuelong Li,et al.  Detection of Co-salient Objects by Looking Deep and Wide , 2016, International Journal of Computer Vision.

[33]  Changsheng Xu,et al.  Multimodal Spatio-Temporal Theme Modeling for Landmark Analysis , 2014, IEEE MultiMedia.

[34]  Xueming Qian,et al.  What Is Happening in the Video? —Annotate Video by Sentence , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Thomas Sikora,et al.  How Spatial Segmentation improves the Multimodal Geo-Tagging , 2012, MediaEval.

[37]  Jon M. Kleinberg,et al.  Mapping the world's photos , 2009, WWW '09.

[38]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[39]  Steven Schockaert,et al.  Ghent University at the 2010 placing task , 2010 .

[40]  Xiaochun Cao,et al.  Self-Adaptively Weighted Co-Saliency Detection via Rank Constraint , 2014, IEEE Transactions on Image Processing.

[41]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[42]  Alexei A. Efros,et al.  IM2GPS: estimating geographic information from a single image , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Xueming Qian,et al.  Scalable Mobile Image Retrieval by Exploring Contextual Saliency , 2015, IEEE Transactions on Image Processing.

[44]  Tieniu Tan,et al.  Deep semantic ranking based hashing for multi-label image retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Xiaoqiang Lu,et al.  Scene Recognition by Manifold Regularized Deep Learning Architecture , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[46]  Shuicheng Yan,et al.  Towards efficient sparse coding for scalable image annotation , 2013, ACM Multimedia.

[47]  Xueming Qian,et al.  Service Rating Prediction by Exploring Social Mobile Users’ Geographical Locations , 2017, IEEE Transactions on Big Data.

[48]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[49]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[50]  Michael Isard,et al.  Bundling features for large scale partial-duplicate web image search , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Yuting Zhang,et al.  Sketch-Based Image Retrieval by Salient Contour Reinforcement , 2016, IEEE Transactions on Multimedia.

[52]  Gang Hua,et al.  Descriptive visual words and visual phrases for image applications , 2009, ACM Multimedia.

[53]  Zhiwei Li,et al.  Contextual synonym dictionary for visual object retrieval , 2011, ACM Multimedia.

[54]  Bo Xu,et al.  Effective near-duplicate image retrieval with image-specific visual phrase selection , 2012, 2012 19th IEEE International Conference on Image Processing.

[55]  Xueming Qian,et al.  Image Taken Place Estimation via Geometric Constrained Spatial Layer Matching , 2015, MMM.

[56]  Xuelong Li,et al.  Surveillance Video Synopsis via Scaling Down Objects , 2016, IEEE Transactions on Image Processing.

[57]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[58]  Ricardo da Silva Torres,et al.  Visual word spatial arrangement for image retrieval and classification , 2014, Pattern Recognit..

[59]  Pingkun Yan,et al.  Image Super-Resolution Via Double Sparsity Regularized Manifold Learning , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[60]  Lei Zhang,et al.  Bit-Scalable Deep Hashing With Regularized Similarity Learning for Image Retrieval and Person Re-Identification , 2015, IEEE Transactions on Image Processing.

[61]  Xueming Qian,et al.  HWVP: hierarchical wavelet packet descriptors and their applications in scene categorization and semantic concept retrieval , 2012, Multimedia Tools and Applications.

[62]  Pingkun Yan,et al.  Alternatively Constrained Dictionary Learning For Image Superresolution , 2014, IEEE Transactions on Cybernetics.

[63]  Xuelong Li,et al.  Semi-Supervised Multitask Learning for Scene Recognition , 2015, IEEE Transactions on Cybernetics.

[64]  Xuelong Li,et al.  Multiresolution Imaging , 2014, IEEE Transactions on Cybernetics.

[65]  Xueming Qian,et al.  Tag-Based Image Search by Social Re-ranking , 2016, IEEE Transactions on Multimedia.

[66]  Tao Mei,et al.  Learning salient visual word for scalable mobile image retrieval , 2015, Pattern Recognit..

[67]  Michael Isard,et al.  Object retrieval with large vocabularies and fast spatial matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[68]  Michael Isard,et al.  Lost in quantization: Improving particular object retrieval in large scale image databases , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[69]  Jiebo Luo,et al.  Beyond GPS: determining the camera viewing direction of a geotagged image , 2010, ACM Multimedia.

[70]  Xiangtao Zheng,et al.  Joint Dictionary Learning for Multispectral Change Detection , 2017, IEEE Transactions on Cybernetics.

[71]  Changsheng Xu,et al.  Interaction Design for Mobile Visual Search , 2013, IEEE Transactions on Multimedia.

[72]  Gang Hua,et al.  Building contextual visual vocabulary for large-scale image applications , 2010, ACM Multimedia.

[73]  Ming Yang,et al.  Discovery of Collocation Patterns: from Visual Words to Visual Phrases , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[74]  Xueming Qian,et al.  Improved image GPS location estimation by mining salient features , 2015, Signal Process. Image Commun..

[75]  Hanjiang Lai,et al.  Supervised Hashing for Image Retrieval via Image Representation Learning , 2014, AAAI.

[76]  Daniel P. Huttenlocher,et al.  Landmark classification in large-scale image collections , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[77]  Marc Pollefeys,et al.  Leveraging 3D City Models for Rotation Invariant Place-of-Interest Recognition , 2011, International Journal of Computer Vision.

[78]  Yi Yang,et al.  A Multimedia Retrieval Framework Based on Semi-Supervised Ranking and Relevance Feedback , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Wen Gao,et al.  Towards compact topical descriptors , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[80]  Xuelong Li,et al.  Latent Semantic Minimal Hashing for Image Retrieval , 2017, IEEE Transactions on Image Processing.

[81]  Jen-Hao Hsiao,et al.  Deep learning of binary hash codes for fast image retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).