Automated Lifelog Moment Retrieval based on Image Segmentation and Similarity Scores

In our working notes we discuss our proposed strategies for the ImageCLEFlifelog LMRT task. We used word vectors to calculate similarity scores between queries and images. To extract important moments and reduce the image amount we used image segmentation based on histograms. We enriched the given data with concepts from pretrained models and got twelve concept types for which similarity scores were calculated and accounted. Furthermore, we used tree boosting as a predictive approach. Our highest F1@10 on the training queries was 27.41% and for the test queries we obtained a maximal F1@10 of 11.70%. All of our models were applicable to generic queries in a fully automated manner.

[1]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[2]  Bogdan Ionescu,et al.  Multimedia Lab @ ImageCLEF 2018 Lifelog Moment Retrieval Task , 2018, CLEF.

[3]  Carlos R. del-Blanco,et al.  ImageCLEF 2019: Multimedia Retrieval in Medicine, Lifelogging, Security and Nature , 2019, CLEF.

[4]  Alan F. Smeaton,et al.  LifeLogging: Personal Big Data , 2014, Found. Trends Inf. Retr..

[5]  Hussein Hussein,et al.  Technische Universität Chemnitz at TRECVID Instance Search 2015 , 2014, TRECVID.

[6]  Wei-Hao Lin,et al.  Structuring continuous video recordings of everyday life using time-constrained clustering , 2006, Electronic Imaging.

[7]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[8]  Anind K. Dey,et al.  Lifelogging memory appliance for people with episodic memory impairment , 2008, UbiComp.

[9]  Michael Riegler,et al.  Overview of ImageCLEFlifelog 2019: Solve My Life Puzzle and Lifelog Moment Retrieval , 2019, CLEF.

[10]  Abigail Sellen,et al.  Beyond total capture , 2010, Commun. ACM.

[11]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[12]  Hussein Hussein,et al.  Acoustic Event Classification Using Convolutional Neural Networks , 2017, GI-Jahrestagung.

[13]  Stefan Kahl,et al.  Technische Universität Chemnitz and Hochschule Mittweida at TRECVID Instance Search 2017 , 2017, TRECVID.

[14]  Petia Radeva,et al.  Visual summary of egocentric photostreams by representative keyframes , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[15]  Stefan Kahl,et al.  Species Prediction based on Environmental Variables using Machine Learning Techniques , 2018, CLEF.

[16]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[17]  Stefan Kahl,et al.  WS34 - Deep Learning in heterogenen Datenbeständen , 2017, GI-Jahrestagung.

[18]  Alan F. Smeaton,et al.  Multimodal Segmentation of Lifelog Data , 2007, RIAO.

[19]  Hervé Glotin,et al.  Overview of BirdCLEF 2018: Monospecies vs. Sundscape Bird Identification , 2018, CLEF.

[20]  Stefan Kahl,et al.  Recognizing Birds from Sound - The 2018 BirdCLEF Baseline System , 2018, ArXiv.

[21]  Michael Riegler,et al.  Overview of ImageCLEFlifelog 2018: Daily Living Understanding and Lifelog Moment Retrieval , 2018, CLEF.

[22]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Michael Riegler,et al.  Overview of ImageCLEFlifelog 2017: Lifelog Retrieval and Summarization , 2017, CLEF.

[25]  Michael Riegler,et al.  Organizer Team at ImageCLEFlifelog 2017: Baseline Approaches for Lifelog Retrieval and Summarization , 2017, CLEF.

[26]  University of Applied Sciences Mittweida and Chemnitz University of Technology at TRECVID Instance Search 2019 , 2020 .

[27]  Stefan Kahl,et al.  Evaluation of CNN-based algorithms for human pose analysis of persons in red carpet scenarios , 2017, GI-Jahrestagung.

[28]  Hervé Glotin,et al.  LifeCLEF 2019: Biodiversity Identification and Prediction Challenges , 2019, ECIR.

[29]  Hsin-Hsi Chen,et al.  Visual Concept Selection with Textual Knowledge for Understanding Activities of Daily Living and Life Moment Retrieval , 2018, CLEF.

[30]  Chokri Ben Amar,et al.  Regim Lab Team at ImageCLEF Lifelog Moment Retrieval Task 2018 , 2018, CLEF.