Lifelog Moment Retrieval with Self-Attention based Joint Embedding Model

With the swift growth of technology, personal devices like cameras or healthcare sensors are more and more approachable, and many people use these devices to record their daily lives. So there is an increasing need for exploiting that enormous amount of data to understand more about how people live their lives. Thus, we introduce a novel interactive system to retrieve specific moments utilizing textbased queries. We propose Self-Attention based Joint Embedding Model (SAJEM) for that purpose. In our proposed method, we first extract visual and text features, then map them to a single common space, and calculate cosine distance for ranking. Besides, our system has two more auxiliary components using ResNet152 features and metadata of images to help users extend their query results. We also design a web application with an easy-to-use user interface to visualize and retrieve lifelog data. With this solution, we achieve the first rank in Lifelog Moment Retrieval task of ImageCLEF Lifelog 2020 with F1@10 score of 0.811.