A Multi-level Interactive Lifelog Search Engine with User Feedback

With the rise of portable wearable devices, it is easier for users to save their lifelog data. As lifelog is usually disorganized with multi-modal information (even noisy sometimes), an interactive search engine is crucial for users to review and explore their lifelog. Unlike traditional search engines, lifelog search includes multi-modality information of images, text and other data from sensors, which brings challenges to data arrangement and search. Accordingly, users' information need is also multi-level. Hence, a single interaction mechanism may not be able to satisfy users' requirements. As the data set is highly personalized, interaction and feedback from users should also be considered in the search engine. Therefore, in this paper we present an interactive multi-modality lifelog search engine to help users manage and find lifelog data. To this end, lifelog data is clustered and processed in multi-level processing. Then, we build an interactive search engine, includingtext as query, image as query, andtimeline view modules. Besides, the system is able to adopt user feedback mechanisms in multi-round queries. Our system shows promising experimental results on LSC'20 dataset and development topics. The text-based search module gives correct results on more than 60% of the development topics at LSC'20.

[1]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[2]  Fuhui Long,et al.  Fundamentals of Content-Based Image Retrieval , 2003 .

[3]  Rob Kitchin,et al.  ‘Outlines of a World Coming into Existence’: Pervasive Computing and the Ethics of Forgetting , 2007 .

[4]  Rogério Schmidt Feris,et al.  Dialog-based Interactive Image Retrieval , 2018, NeurIPS.

[5]  Klaus Schöffmann,et al.  LSE2018 Panel - Challenges of Lifelog Search and Access , 2018, LSC@ICMR.

[6]  Yiqun Liu,et al.  A Two-Level Lifelog Search Engine at the LSC 2019 , 2019, LSC '19.

[7]  Minh-Triet Tran,et al.  Introduction to the Third Annual Lifelog Search Challenge (LSC'20) , 2020, ICMR.

[8]  Heiko Schuldt,et al.  Retrieval of Structured and Unstructured Data with vitrivr , 2019, LSC@ICMR.

[9]  Jakub Lokoc,et al.  Enhanced VIRET Tool for Lifelog Data , 2019, LSC '19.

[10]  Klaus Schöffmann,et al.  lifeXplore at the Lifelog Search Challenge 2018 , 2018, LSC@ICMR.

[11]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Vinh-Tiep Nguyen,et al.  Smart Lifelog Retrieval System with Habit-based Concepts and Moment Visualization , 2019, LSC '19.

[13]  Marcel Worring,et al.  Exquisitor at the Lifelog Search Challenge 2019 , 2019, LSC@ICMR.

[14]  Nuno Vasconcelos,et al.  Learning from User Feedback in Image Retrieval Systems , 1999, NIPS.