MyEachtra: Event-Based Interactive Lifelog Retrieval System for LSC’23

Retrieval is a fundamental challenge within the research community of lifelog and the Lifelog Search Challenge (LSC) has been an important annual benchmarking activity for interactive lifelog retrieval systems since 2018. This paper proposes MyEachtra (/mai-AK-truh/), a system designed for the upcoming LSC’23 workshop. Improved upon MyScéal, which was the top performing system from LSC’20 to LSC’22, MyEachtra includes modifications to address the challenges of non-owner user understanding of lifelog contexts and open-ended lifelog question answering. Specifically, MyEachtra shifts the focus from images to events as retrieval units. Events are segmented using location metadata as well as visual and time differences between successive images. A pilot study on different approaches to aggregate images into events was conducted to test the automatic performance of the system, which showed promising results. For known-item queries, showing only the top 3 events proved to be adequate to find relevant images. However, future evaluation of the performance for ad-hoc and question-answering queries is necessary for a complete analysis of the MyEachtra.

[1]  Duc Tien Dang Nguyen,et al.  Introduction to the Sixth Annual Lifelog Search Challenge, LSC’23 , 2023, ICMR.

[2]  Yonghui Wu,et al.  VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners , 2022, 2212.04979.

[3]  Ludwig Schmidt,et al.  LAION-5B: An open large-scale dataset for training next generation image-text models , 2022, NeurIPS.

[4]  C. Gurrin,et al.  E-Myscéal: Embedding-based Interactive Lifelog Retrieval System for LSC'22 , 2022, LSC@ICMR.

[5]  C. Gurrin,et al.  LifeSeeker 4.0: An Interactive Lifelog Search Engine for LSC'22 , 2022, LSC@ICMR.

[6]  Yvette Graham,et al.  Memento 2.0: An Improved Lifelog Search Engine for LSC'22 , 2022, LSC@ICMR.

[7]  C. Gurrin,et al.  Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22 , 2022, ICMR.

[8]  C. Gurrin,et al.  Flexible Interactive Retrieval SysTem 3.0 for Visual Lifelog Exploration at LSC 2022 , 2022, LSC@ICMR.

[9]  A. Neves,et al.  MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2022 , 2022, LSC@ICMR.

[10]  C. Schmid,et al.  Zero-Shot Video Question Answering via Frozen Bidirectional Language Models , 2022, NeurIPS.

[11]  Andrew Zisserman,et al.  A CLIP-Hitchhiker's Guide to Long Video Retrieval , 2022, ArXiv.

[12]  Zirui Wang,et al.  CoCa: Contrastive Captioners are Image-Text Foundation Models , 2022, Trans. Mach. Learn. Res..

[13]  Yi Yang,et al.  CenterCLIP: Token Clustering for Efficient Text-Video Retrieval , 2022, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[14]  Weidi Xie,et al.  Prompting Visual-Language Models for Efficient Video Understanding , 2021, ECCV.

[15]  C. Gurrin,et al.  Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC'21 , 2021, LSC@ICMR.

[16]  Nan Duan,et al.  CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval , 2021, Neurocomputing.

[17]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[18]  Jes'us Andr'es Portillo-Quintero,et al.  A Straightforward Framework For Video Retrieval Using CLIP , 2021, MCPR.

[19]  Quoc V. Le,et al.  Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision , 2021, ICML.

[20]  Nguyen Thanh Binh,et al.  Myscéal: An Experimental Interactive Lifelog Retrieval System for LSC'20 , 2020, LSC@ICMR.

[21]  Minh-Triet Tran,et al.  Introduction to the Third Annual Lifelog Search Challenge (LSC'20) , 2020, ICMR.

[22]  Heiko Schuldt,et al.  Retrieval of Structured and Unstructured Data with vitrivr , 2019, LSC@ICMR.

[23]  Minh-Triet Tran,et al.  [Invited papers] Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018) , 2019, ITE Transactions on Media Technology and Applications.

[24]  Cathal Gurrin,et al.  Virtual Reality Lifelog Explorer: Lifelog Search Challenge at ACM ICMR 2018 , 2018, LSC@ICMR.

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[27]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[28]  Hans-Peter Kriegel,et al.  OPTICS: ordering points to identify the clustering structure , 1999, SIGMOD '99.

[29]  C. Gurrin,et al.  Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021 , 2023, IEEE Access.

[30]  C. Gurrin,et al.  VAISL: Visual-Aware Identification of Semantic Locations in Lifelog , 2023, MMM.

[31]  C. Gurrin,et al.  LLQA - Lifelog Question Answering Dataset , 2022, MMM.

[32]  C. Gurrin,et al.  Overview of the NTCIR-16 Lifelog-4 Task , 2022 .