Affective computing for wearable diary and lifelogging systems: An overview

Diaries have transformed over the last decade. Originally in handwritten text format, photo albums and visual diaries became popular as photography became commonly available. Traditionally intended to remain private, the dimension of audience was added in blogs along with Internet communication. However, humans have a limited capacity to record their lives. Lifelogs overcome this limit and collect and store a person’s personal information digitally. This can be done by recording all computer and cell phone activity and mobile context (e.g. GPS), but also adding multiple wearable sensors such as “always-on” cameras or bio-sensors. The enormous amounts of data created needs to be processed in order to be made useful to humans. In this paper, we review state-of-the-art literature on wearable diaries and lifelogging systems, and discuss the key issues and main challenges. 1 Diaries, Blogs and Lifelogs Humans record their history, from global events that move the world to small everyday life events that they consider important. In this paper, we review the current trends of personal record keeping. We focus on approaches using wearable sensors to passively capture data. First, forms of personal record keeping are discussed, especially how diaries have changed over time, becoming increasingly multimodal. Section 2 describes key issues and main challenges of building modern diary systems. In Section 3 we present selected projects which offer unique diary solutions. We conclude with thoughts on future challenges are offered. 1.1. Traditional diary A diary is a sequence of entries arranged chronologically, created to report on what has happened over the course of a period of time. Personal diaries usually include the writer’s thoughts and feelings. Originally handwritten in the form of books, the diary transformed from paper to electronic formats. Along with new media, new forms of diaries also developed. 1.1.1 Visual diary Since photography became commonly available, photos were added to the text in the diary to illustrate events. In some cases the emphasis shifted towards photography altogether and it became common to create photo albums with short text comments. Photos simplified the record keeping and added a further dimension to the records that made the memories seem more real [26]. Sontag [26] goes even further and says that it is the photographic capture of reality that gives us the feeling of the realness of our lives, it helps us reconstruct our personal history. We don’t believe our perception until a photo confirms it [26] as is also illustrated by the popular Internet-phrase “pics or it didn’t happen” ∗This work was supported by FFG, project no. 830043 demanding photographic proof of an unbelievable story. Artists use visual diaries to sketch drawings of their ideas and to collect images or other media. They serve them as inspiration and as a means to reflect on their artistic growth [1]. In travel diaries the author might even add small souvenirs, brochures, postcards or other nostalgic items to the diary. 1.1.2 Blogs The most distinct and today probably also most popular form of diary are blogs. While the “paper version” was traditionally intended to remain private, this aspect widely changed as people adopted the new medium to chronicle their lives with the additional dimension of audience. Although the reasons given by bloggers such as documenting my life, expressing opinions, letting off steam (shouts, feelings and thoughts), inspiration (“to write is to think”), and building communities, the bloggers are conscious about their audience and censor the information they publish [21]. Admittedly, the entries serve rather for self-presentation and narcissism than for creating an extensive digital memory [21]. Still, the main point of a traditional diary (public or private) is that the content is actively generated by the user. However, documenting a life takes discipline and effort and, hence, the amount of information in the traditional diary is constrained either by the capacity of the medium or the writer. This limit forces the individual to filter the content he/she writes down. 1.2. LifeLog This issue of necessity (and difficulty) to limit information that can be preserved in a diary by a single human is being addressed by new technologies. Technological advancements today make it possible to explore new ways and possibilities to capture, collect and store information. First envisioned by Vannevar Bush in 1945 [5], a LifeLog presents the notion to capture and store a whole lifetime of a person’s personal information digitally, so that it can be retrieved whenever needed. In 2001, this idea was revived by an experiment of Gordon Bell [2], who scanned all of his paperwork, photographs, medical records and other personal data. The initial focus for lifelogging was on desktop applications only, however, it has shifted towards mobile access and capture. Mobile devices are capable of more than just storing they capture data passively, without the user’s conscious initiative. Lifelog systems collect a variety of signals subsets of the following data categories (as described in [7]): Passive visual capture: Wearable “always-on” cameras automatically take images or videos. Biometrics: Wearable sensors measuring bio-signals, such as heart rate, galvanic skin response, skin temperature or body motion. Mobile context: Cell phones can provide information such as location cues in the form of GPS data, wireless network presence and GSM location data. Co-present Bluetooth devices may indicate people present nearby. Mobile activity: Call logs, SMSs, even email-logs, activity on the web and social networks sites can be gathered from mobile phones. Desktop/laptop computer activity: Basically all PC/laptop activity of the user can be monitored, the time and duration of each task measured, documents saved, etc. Active capture: Indirect (writing blogs is monitored as computer activity) or direct (add photos, write comments and annotate the lifelog’s content) capture. Such technologies, however, are not limited to the obvious use of personal reminiscence only. Various applications featuring at least a subset of these possibilities have been used, e.g. in medical and therapeutic solutions, security enhancement, or to encourage self-reflection.