Computer Vision for Lifelogging: Characterizing Everyday Activities Based on Visual Semantics

Abstract The rapid development of mobile devices capable of sensing our interactions with the environment has made it possible to assist humans in daily living, for example helping patients with cognitive impairment or providing customized food intake plans for patients with obesity, etc. All of this can be achieved through the passive gathering of detailed records of everyday behavior known as lifelogging. For example, the widely adopted smart mobile phones and newly emerging consumer wearable devices like Google Glass, Baidu Eye, and Narrative Clip are usually embedded with rich sensing capabilities including a camera, accelerometer, GPS, digital compass, etc., which can help to capture daily activity unobtrusively. Among these heterogeneous sensor readings, visual media contain more semantics to assist in characterizing everyday activities, and visual lifelogging is a class of personal sensing that employs wearable cameras to capture images or video sequences of everyday activities. This chapter will focus on the most recent research methods used to understand visual lifelogs, including semantic annotations of visual concepts, use of contextual semantics, recognition of activities, visualization of activities, etc. We also discuss some research challenges that indicate potential directions for future research. This chapter is intended to support readers in the area of assistive living using wearable sensing and computer vision for lifelogging, and human behavior researchers aiming at behavioral analysis based on visual understanding.