Semantic Guided Interactive Image Retrieval for plant identification

Abstract A lot of images are currently generated in many domains, requiring specialized knowledge of identification and analysis. From one standpoint, many advances have been accomplished in the development of image retrieval techniques based on visual image properties. However, the semantic gap between low-level features and high-level concepts still represents a challenging scenario. On another standpoint, knowledge has also been structured in many fields by ontologies. A promising solution for bridging the semantic gap consists in combining the information from low-level features with semantic knowledge. This work proposes a novel graph-based approach denominated Semantic Interactive Image Retrieval (SIIR) capable of combining Content Based Image Retrieval (CBIR), unsupervised learning, ontology techniques and interactive retrieval. To the best of our knowledge, there is no approach in the literature that combines those diverse techniques like SIIR. The proposed approach supports expert identification tasks, such as the biologist’s role in plant identification of Angiosperm families. Since the system exploits information from different sources as visual content, ontology, and user interactions, the user efforts required are drastically reduced. For the semantic model, we developed a domain ontology which represents the plant properties and structures, relating features from Angiosperm families. A novel graph-based approach is proposed for combining the semantic information and the visual retrieval results. A bipartite and a discriminative attribute graph allow a semantic selection of the most discriminative attributes for plant identification tasks. The selected attributes are used for formulating a question to the user. The system updates similarity information among images based on the user’s answer, thus improving the retrieval effectiveness and reducing the user’s efforts required for identification tasks. The proposed method was evaluated on the popular Oxford Flowers 17 and 102 Classes datasets, yielding highly effective results in both datasets when compared to other approaches. For example, the first five retrieved images for 17 classes achieve a retrieval precision of 97.07% and for 102 classes, 91.33%.

[1]  Jian Guan,et al.  Learning user intention in relevance feedback using optimization , 2007, MIR '07.

[2]  Lee W. Lacy.  OWL: Representing Information Using the Web Ontology Language , 2006 .

[3]  Malay Kumar Kundu,et al.  A graph-based relevance feedback mechanism in content-based image retrieval , 2015, Knowl. Based Syst..

[4]  Mario A. Nascimento,et al.  A compact and efficient image retrieval approach based on border/interior pixel classification , 2002, CIKM '02.

[5]  Umar Manzoor,et al.  Ontology-Based Clinical Decision Support System for Predicting High-Risk Pregnant Woman , 2015 .

[6]  Daniel Carlos Guimarães Pedronette,et al.  Unsupervised manifold learning using Reciprocal kNN Graphs in image re-ranking and rank aggregation tasks , 2014, Image Vis. Comput..

[7]  Fahad Shahbaz Khan,et al.  Portmanteau Vocabularies for Multi-Cue Image Representation , 2011, NIPS.

[8]  Wei Liu,et al.  Semi-supervised distance metric learning for collaborative image retrieval and clustering , 2010, ACM Trans. Multim. Comput. Commun. Appl..

[9]  Andrew Zisserman,et al.  BiCoS: A Bi-level co-segmentation method for image classification , 2011, 2011 International Conference on Computer Vision.

[10]  L. Durieux,et al.  Advances in Geographic Object-Based Image Analysis with ontologies: A review of main contributions and limitations from a remote sensing perspective , 2013 .

[11]  Tao Qin,et al.  Supervised rank aggregation , 2007, WWW '07.

[12]  Andrew Zisserman,et al.  A Visual Vocabulary for Flower Classification , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[13]  Andréa Britto Mattos,et al.  A mission-oriented citizen science platform for efficient flower classification based on combination of feature descriptors , 2014, EMR@ICMR.

[14]  Thomas R. Gruber,et al.  A translation approach to portable ontology specifications , 1993, Knowl. Acquis..

[15]  Francesc J. Ferri,et al.  An improved distance-based relevance feedback strategy for image retrieval , 2013, Image Vis. Comput..

[16]  Bernt Schiele,et al.  International Journal of Computer Vision manuscript No. (will be inserted by the editor) Semantic Modeling of Natural Scenes for Content-Based Image Retrieval , 2022 .

[17]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Mathias Lux,et al.  Lire: lucene image retrieval: an extensible java CBIR library , 2008, ACM Multimedia.

[19]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[20]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[21]  N. Guarino,et al.  Formal Ontology in Information Systems: Proceedings of the 1st International Conference June 6-8, 1998, Trento, Italy , 1998 .

[22]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[23]  Nicu Sebe,et al.  Content-based multimedia information retrieval: State of the art and challenges , 2006, TOMCCAP.

[24]  Andrew Zisserman,et al.  Automated Flower Classification over a Large Number of Classes , 2008, 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing.

[25]  Garrison W. Cottrell,et al.  Robust classification of objects, faces, and flowers using natural image statistics , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Pritee Khanna,et al.  An effective use of adaptive combination of visual features to retrieve image semantics from a hierarchical image database , 2015, J. Vis. Commun. Image Represent..

[27]  Haibin Ling,et al.  Image Classification Using Multiscale Information Fusion Based on Saliency Driven Nonlinear Diffusion Filtering , 2014, IEEE Transactions on Image Processing.

[28]  Asunción Gómez-Pérez,et al.  METHONTOLOGY: From Ontological Art Towards Ontological Engineering , 1997, AAAI 1997.

[29]  Ricardo da Silva Torres,et al.  Image re-ranking and rank aggregation based on similarity of ranked lists , 2013, Pattern Recognit..

[30]  Mathias Lux.  LIRE: open source image retrieval in Java , 2013, MM '13.

[31]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[32]  Song Bai,et al.  Beyond diffusion process: Neighbor set similarity for fast re-ranking , 2015, Inf. Sci..

[33]  Jing Huang,et al.  Image indexing using color correlograms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Vishwanath Reddy Keshi Reddy,et al.  Image Retrieval Using a Combination of Keywords and Image Features , 2008 .

[35]  Song Bai,et al.  Sparse Contextual Activation for Efficient Visual Re-Ranking , 2016, IEEE Transactions on Image Processing.

[36]  Bart Thomee,et al.  Interactive search in image retrieval: a survey , 2012, International Journal of Multimedia Information Retrieval.

[37]  Giorgio Giacinto,et al.  A nearest-neighbor approach to relevance feedback in content based image retrieval , 2007, CIVR '07.

[38]  Barry Smith,et al.  Ontologies as integrative tools for plant science. , 2012, American journal of botany.

[39]  Manik Varma,et al.  Learning The Discriminative Power-Invariance Trade-Off , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[40]  Nozha Boujemaa,et al.  Pl@ntNet mobile app , 2013, ACM Multimedia.

[41]  Daniel L. Rubin,et al.  On combining image-based and ontological semantic dissimilarities for medical image retrieval applications , 2014, Medical Image Anal..

[42]  Dzulkifli Mohamad,et al.  Semantic gap in CBIR: automatic objects spatial relationships semantic extraction and representation , 2010 .

[43]  Paul Wing Hing Kwan,et al.  A knowledge-based Decision Support System for adaptive fingerprint identification that uses relevance feedback , 2015, Knowl. Based Syst..

[44]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[45]  Gözde B. Ünal,et al.  Plant image retrieval using color and texture features , 2009, 2009 24th International Symposium on Computer and Information Sciences.

[46]  Holger Knublauch,et al.  The Protégé OWL Plugin: An Open Development Environment for Semantic Web Applications , 2004, SEMWEB.

[47]  Satoshi Ito,et al.  Object Classification Using Heterogeneous Co-occurrence Features , 2010, ECCV.

[48]  Carlos Caballero,et al.  Plant species identification using leaf image retrieval , 2010, CIVR '10.

[49]  Daniel Carlos Guimarães Pedronette,et al.  Unsupervised manifold learning through reciprocal kNN graph and Connected Components for image retrieval tasks , 2018, Pattern Recognit..

[50]  Lei Zhang,et al.  A Unified Relevance Feedback Framework for Web Image Retrieval , 2009, IEEE Transactions on Image Processing.

[51]  Tat-Seng Chua,et al.  A bootstrapping approach to annotating large image collection , 2003, MIR '03.

[52]  LiuWei,et al.  Semi-supervised distance metric learning for collaborative image retrieval and clustering , 2010 .

[53]  Fahad Shahbaz Khan,et al.  Top-down color attention for object recognition , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[54]  Shenghuo Zhu,et al.  Image segmentation for large-scale subcategory flower recognition , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[55]  Xiaoqin Zhang,et al.  Use bin-ratio information for category and scene classification , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.