Flexible Interactive Retrieval SysTem 2.0 for Visual Lifelog Exploration at LSC 2021

With a huge collection of photos and video clips, it is essential to provide an efficient and easy-to-use system for users to retrieve moments of interest with a wide variation of query types. This motivates us to develop and upgrade our flexible interactive retrieval system for visual lifelog exploration. In this paper, we briefly introduce version 2 of our system with the following main features. Our system supports multiple modalities for interaction and query processing, including visual query by meta-data, text query and visual information matching based on a joint embedding model, scene clustering based on visual and location information, flexible temporal event navigation, and query expansion with visual examples. With the flexibility in system architecture, we expect our system can easily integrate new modules to enhance its functionalities.

[1]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Hao Chen,et al.  ABCNet: Real-Time Scene Text Spotting With Adaptive Bezier-Curve Network , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Minh-Triet Tran,et al.  [Invited papers] Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018) , 2019, ITE Transactions on Media Technology and Applications.

[4]  Vinh-Tiep Nguyen,et al.  Smart Lifelog Retrieval System with Habit-based Concepts and Moment Visualization , 2019, LSC '19.

[5]  Minh-Triet Tran,et al.  FIRST - Flexible Interactive Retrieval SysTem for Visual Lifelog Exploration at LSC 2020 , 2020, LSC@ICMR.

[6]  Jianfeng Gao,et al.  Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks , 2020, ECCV.

[7]  Minh-Triet Tran,et al.  Introduction to the Fourth Annual Lifelog Search Challenge, LSC'21 , 2021, ICMR.

[8]  Minh-Triet Tran,et al.  Lifelog Moment Retrieval with Self-Attention based Joint Embedding Model , 2020, CLEF.

[9]  Heiko Schuldt,et al.  Interactive Lifelog Retrieval with vitrivr , 2020, LSC@ICMR.

[10]  Minh-Triet Tran,et al.  LifeSeeker 2.0: Interactive Lifelog Search Engine at LSC 2020 , 2020, LSC@ICMR.

[11]  Yu Cheng,et al.  UNITER: UNiversal Image-TExt Representation Learning , 2019, ECCV.

[12]  Luca Rossetto,et al.  LifeGraph: A Knowledge Graph for Lifelogs , 2020, LSC@ICMR.

[13]  Rami Albatal,et al.  Overview of the NTCIR-14 Lifelog-3 task , 2019 .

[14]  Michael Riegler,et al.  Overview of ImageCLEFlifelog 2019: Solve My Life Puzzle and Lifelog Moment Retrieval , 2019, CLEF.

[15]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[16]  Gregor Kovalčík,et al.  VIRET Tool with Advanced Visual Browsing and Feedback , 2020, LSC@ICMR.

[17]  Vinh-Tiep Nguyen,et al.  Lifelog Moment Retrieval with Advanced Semantic Extraction and Flexible Moment Visualization for Exploration , 2019, CLEF.

[18]  Marcel Worring,et al.  Exquisitor at the Lifelog Search Challenge 2020 , 2020, LSC@ICMR.

[19]  Xi Chen,et al.  Stacked Cross Attention for Image-Text Matching , 2018, ECCV.

[20]  Cathal Gurrin,et al.  Voxento: A Prototype Voice-controlled Interactive Search Engine for Lifelogs , 2020, LSC@ICMR.

[21]  Yun Fu,et al.  Visual Semantic Reasoning for Image-Text Matching , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[22]  Cathal Gurrin,et al.  VRLE: Lifelog Interaction Prototype in Virtual Reality: Lifelog Search Challenge at ACM ICMR 2020 , 2020, LSC@ICMR.

[23]  Seong Joon Oh,et al.  Probabilistic Embeddings for Cross-Modal Retrieval , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Minh-Triet Tran,et al.  Overview of ImageCLEF Lifelog 2020: Lifelog Moment Retrieval and Sport Performance Lifelog , 2020, CLEF.

[25]  Minh-Triet Tran,et al.  Introduction to the Third Annual Lifelog Search Challenge (LSC'20) , 2020, ICMR.

[26]  Michael Riegler,et al.  Paper Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge ( LSC 2018 ) , 2019 .

[27]  Jakub Lokoč,et al.  SOMHunter for Lifelog Search , 2020, LSC@ICMR.

[28]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[29]  Quoc V. Le,et al.  EfficientDet: Scalable and Efficient Object Detection , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  David J. Fleet,et al.  VSE++: Improving Visual-Semantic Embeddings with Hard Negatives , 2017, BMVC.

[31]  Nguyen Thanh Binh,et al.  Myscéal: An Experimental Interactive Lifelog Retrieval System for LSC'20 , 2020, LSC@ICMR.