Approaches for Event Segmentation of Visual Lifelog Data

A personal visual lifelog can be considered to be a human memory augmentation tool and in recent years we have noticed an increased interest in the topic of lifelogging both in academic research and from industry practitioners. In this preliminary work, we explore the concept of event segmentation of visual lifelog data. Lifelog data, by its nature is continual and streams of multimodal data can easy run into thousands of wearable camera images per day, along with a significant number of other sensor sources. In this paper, we present two new approaches to event segmentation and compare them against pre-existing approaches in a user experiment with ten users. We show that our approaches based on visual concepts occurrence and image categorization perform better than the pre-existing approaches. We finalize the paper with a suggestion for next steps for the research community.

[1]  Geoffrey Zweig,et al.  From captions to visual concepts and back , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[3]  Alan F. Smeaton,et al.  Architecture and challenges of maintaining a large-scale, context-aware human digital memory , 2008 .

[4]  Vannevar Bush,et al.  As we may think , 1945, INTR.

[5]  Alan F. Smeaton,et al.  Automatically Segmenting LifeLog Data into Events , 2008, 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services.

[6]  G. Bell,et al.  A digital life , 2007 .

[7]  Alan F. Smeaton,et al.  Using bluetooth and GPS metadata to measure event similarity in SenseCam Images , 2007 .

[8]  Petia Radeva,et al.  Visual summary of egocentric photostreams by representative keyframes , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[9]  Zhen Li,et al.  Daily life event segmentation for lifestyle evaluation based on multi-sensor data recorded by a wearable device , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[10]  Debasis Ganguly,et al.  Segmenting and Summarizing General Events in a Long-Term Lifelog , 2011 .

[11]  Noel E. O'Connor,et al.  LoggerMan, a Comprehensive Logging and Visualization Tool to Capture Computer Usage , 2016, MMM.

[12]  Alan F. Smeaton,et al.  Multimodal Segmentation of Lifelog Data , 2007, RIAO.

[13]  Ullas Gargi,et al.  Modeling and clustering of photo capture streams , 2003, MIR '03.

[14]  Rami Albatal,et al.  NTCIR Lifelog: The First Test Collection for Lifelog Research , 2016, SIGIR.

[15]  Gordon Bell,et al.  MyLifeBits: a personal database for everything , 2006, CACM.

[16]  Alan F. Smeaton,et al.  LifeLogging: Personal Big Data , 2014, Found. Trends Inf. Retr..

[17]  Jeffrey M. Zacks,et al.  Human brain activity time-locked to perceptual event boundaries , 2001, Nature Neuroscience.