Memento 3.0: An Enhanced Lifelog Search Engine for LSC’23

In this work, we present our system Memento 3.0 for participation in the Lifelog Search Challenge 2023, which is a successor to the previous 2 iterations of our system called Memento 1.0 [1] and Memento 2.0 [2]. Memento 3.0 employs image-text embeddings derived from OpenAI CLIP models as well as larger OpenCLIP models trained on ∼ 5x more data. Our system also significantly reduces the query processing time by almost 75% when compared to its predecessor systems by employing a cluster-based search technique. We additionally make important updates to the system’s user interface to offer more flexibility to the user and at the same time be better suited to efficiently handle new query types introduced in the Lifelog Search Challenge.

[1]  Duc Tien Dang Nguyen,et al.  Introduction to the Sixth Annual Lifelog Search Challenge, LSC’23 , 2023, ICMR.

[2]  Gabriel Ilharco,et al.  Reproducible Scaling Laws for Contrastive Language-Image Learning , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Ludwig Schmidt,et al.  LAION-5B: An open large-scale dataset for training next generation image-text models , 2022, NeurIPS.

[4]  C. Gurrin,et al.  LifeSeeker 4.0: An Interactive Lifelog Search Engine for LSC'22 , 2022, LSC@ICMR.

[5]  C. Gurrin,et al.  Flexible Interactive Retrieval SysTem 3.0 for Visual Lifelog Exploration at LSC 2022 , 2022, LSC@ICMR.

[6]  H. Schuldt,et al.  vitrivr at the Lifelog Search Challenge 2022 , 2022, LSC@ICMR.

[7]  C. Gurrin,et al.  E-Myscéal: Embedding-based Interactive Lifelog Retrieval System for LSC'22 , 2022, LSC@ICMR.

[8]  H. Schuldt,et al.  Multimodal Interactive Lifelog Retrieval with vitrivr-VR , 2022, LSC@ICMR.

[9]  Yvette Graham,et al.  Memento 2.0: An Improved Lifelog Search Engine for LSC'22 , 2022, LSC@ICMR.

[10]  C. Gurrin,et al.  Introduction to the Fifth Annual Lifelog Search Challenge, LSC'22 , 2022, ICMR.

[11]  C. Gurrin,et al.  Voxento 3.0: A Prototype Voice-Controlled Interactive Search Engine for Lifelog , 2022, LSC@ICMR.

[12]  Andreas Leibetseder,et al.  lifeXplore at the Lifelog Search Challenge 2022 , 2022, LSC@ICMR.

[13]  Minh-Triet Tran,et al.  Introduction to the Fourth Annual Lifelog Search Challenge, LSC'21 , 2021, ICMR.

[14]  C. Gurrin,et al.  Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC'21 , 2021, LSC@ICMR.

[15]  Yvette Graham,et al.  Memento: A Prototype Lifelog Search Engine for LSC'21 , 2021, LSC@ICMR.

[16]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[17]  Quoc V. Le,et al.  Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision , 2021, ICML.

[18]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[19]  Heiko Schuldt,et al.  Cottontail DB: An Open Source Database System for Multimedia Retrieval and Analysis , 2020, ACM Multimedia.

[20]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[21]  Petia Radeva,et al.  Automatic Reminiscence Therapy for Dementia , 2019, ICMR.

[22]  Xirong Li,et al.  W2VV++: Fully Deep Learning for Ad-hoc Video Search , 2019, ACM Multimedia.

[23]  Kenji Karako,et al.  Super-aged society: Constructing an integrated information platform of self-recording lifelogs and medical records to support health care in Japan. , 2019, Bioscience trends.

[24]  Fabio Crestani,et al.  Augmentation of Human Memory: Anticipating Topics that Continue in the Next Meeting , 2018, CHIIR.

[25]  Alejandro Cartas,et al.  Recognizing Activities of Daily Living from Egocentric Images , 2017, IbPRIA.

[26]  Jeff Johnson,et al.  Billion-Scale Similarity Search with GPUs , 2017, IEEE Transactions on Big Data.

[27]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Alan F. Smeaton,et al.  LifeLogging: Personal Big Data , 2014, Found. Trends Inf. Retr..

[30]  C. Gurrin,et al.  VAISL: Visual-Aware Identification of Semantic Locations in Lifelog , 2023, MMM.

[31]  Quoc V. Le,et al.  Combined Scaling for Open-Vocabulary Image Classification , 2022 .

[32]  Oh-Jin Kwon,et al.  Ubiquitous Healthcare System for Analysis of Chronic Patients’ Biological and Lifelog Data , 2018, IEEE Access.