An Experiment in Interactive Retrieval for the Lifelog Moment Retrieval Task at ImageCLEFlifelog2020

The development of technology has led to an increase in mobile devices’ use to keep track of individual daily activities, as known as Lifelogging. Lifelogging has raised many research challenges, one of which is how to retrieve a specific moment in response to a user’s information need. This paper presents an efficient interactive search engine for large multimodal lifelog data which is evaluated in the ImageCLEFlifelog2020 Lifelog Moment Retrieval task (LMRT). The system is the modified version of the Myscéal demonstrator used in the Lifelog Search Challenge 2020, with the addition of visual similarity and a new method of visualising results. In interactive experimentation, our system achieved an F1@10 score of 0.48 in the official submission but can be significantly improved by implementing a number of post-processing steps.

[1]  Michael Riegler,et al.  Overview of ImageCLEFlifelog 2019: Solve My Life Puzzle and Lifelog Moment Retrieval , 2019, CLEF.

[2]  Minh-Triet Tran,et al.  Introduction to the Third Annual Lifelog Search Challenge (LSC'20) , 2020, ICMR.

[3]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[4]  Vinh-Tiep Nguyen,et al.  Lifelog Moment Retrieval with Advanced Semantic Extraction and Flexible Moment Visualization for Exploration , 2019, CLEF.

[5]  Koji Zettsu,et al.  BIDAL@imageCLEFlifelog2019: The Role of Content and Context of Daily Activities in Insights from Lifelogs , 2019, CLEF.

[6]  Michael S. Bernstein,et al.  Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations , 2016, International Journal of Computer Vision.

[7]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Alan F. Smeaton,et al.  LifeLogging: Personal Big Data , 2014, Found. Trends Inf. Retr..

[9]  Rami Albatal,et al.  NTCIR Lifelog: The First Test Collection for Lifelog Research , 2016, SIGIR.

[10]  Hui Wei,et al.  Creating Efficient Visual Codebook Ensembles for Object Categorization , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[12]  Minh-Triet Tran,et al.  Overview of ImageCLEF Lifelog 2020: Lifelog Moment Retrieval and Sport Performance Lifelog , 2020, CLEF.

[13]  Jie Xia,et al.  ZJUTCVR Team at ImageCLEFlifelog2019 Lifelog Moment Retrieval Task , 2019, CLEF.

[14]  Duc-Tien Dang-Nguyen,et al.  A Baseline Search Engine for Personal Life Archives , 2017, LTA@MM.

[15]  Nguyen Thanh Binh,et al.  Myscéal: An Experimental Interactive Lifelog Retrieval System for LSC'20 , 2020, LSC@ICMR.

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Alan F. Smeaton,et al.  Experiences of Aiding Autobiographical Memory Using the SenseCam , 2012, Hum. Comput. Interact..

[18]  Antonio,et al.  ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Internet Applications , 2020, ECIR.

[19]  Minh-Triet Tran,et al.  [Invited papers] Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018) , 2019, ITE Transactions on Media Technology and Applications.

[20]  Petia Radeva,et al.  LEMoRe: A Lifelog Engine for Moments Retrieval at the NTCIR-Lifelog LSAT Task , 2016, NTCIR.