Automatic structuring and augmentation of a lifelog of images

The SenseCam is a passively capturing wearable camera that takes approximately 3,000 images on average per day. This provides a user with an extensive visual diary. Possible applications making use of this device include helping dementia sufferers recall events from short-term memory, and also this device can be used by tourists to maintain an extensive image collection of their trip. However a large image collection will quickly build up, with an average of 1 million images captured each year. This presents a considerable challenge in terms of managing such a large collection and to make it accessible for users. This proposal addresses the problem in 4 steps: 1) Identifying distinct events within the 3,000 images per day 2) Highlighting the most unique of those events 3) Finding similar events to a given event 4) Augmenting the low-quality images from the wearable camera with higher quality images from external sources.

[1]  Mor Naaman,et al.  Towards automatic extraction of event and place semantics from flickr tags , 2007, SIGIR.

[2]  Daniel L. Schacter,et al.  The Seven Sins of Memory: How the Mind Forgets and Remembers , 2001 .

[3]  R. Shepard.  Recognition memory for words, sentences, and pictures , 1967 .

[4]  Zhe Wang,et al.  VFerret: content-based similarity search tool for continuous archived video , 2006, CARPE '06.

[5]  Paul Over,et al.  The TREC-2002 Video Track Report , 2002, TREC.

[6]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[7]  Dave W. Randall,et al.  The past is a different place: they do things differently there , 2008, DIS '08.

[8]  E. Tulving,et al.  Memory Systems 1994 , 1994 .

[9]  Mor Naaman,et al.  Generating diverse and representative image search results for landmarks , 2008, WWW.

[10]  Steve Hodges,et al.  Neuropsychological Rehabilitation , 2013 .

[11]  Mor Naaman,et al.  How flickr helps us make sense of the world: context and content in community-contributed media collections , 2007, ACM Multimedia.

[12]  G. Davies,et al.  Memory in context : context in memory , 1990 .

[13]  M Naveh-Benjamin,et al.  Digit Span, Reading Rate, and Linguistic Relativity , 1986, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[14]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  S. Cherry,et al.  Total recall [life recording software] , 2005, IEEE Spectrum.

[16]  Shahram Izadi,et al.  SenseCam: A Retrospective Memory Aid , 2006, UbiComp.

[17]  Mor Naaman,et al.  Generating summaries and visualization for large collections of geo-referenced photographs , 2006, MIR '06.

[18]  Alan F. Smeaton,et al.  A usage study of retrieval modalities for video shot retrieval , 2006, Inf. Process. Manag..

[19]  Jonathan Foote,et al.  Discriminative techniques for keyframe selection , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[20]  Anind K. Dey,et al.  Providing good memory cues for people with episodic memory impairment , 2007, Assets '07.

[21]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[22]  N. Nikolaidis,et al.  Video shot detection and condensed representation. a review , 2006, IEEE Signal Processing Magazine.

[23]  Ting Liu,et al.  Clustering Billions of Images with Large Scale Nearest Neighbor Search , 2007, 2007 IEEE Workshop on Applications of Computer Vision (WACV '07).

[24]  Richard C. Atkinson,et al.  Human Memory: A Proposed System and its Control Processes , 1968, Psychology of Learning and Motivation.

[25]  John D E Gabrieli,et al.  Sex differences in the neural basis of emotional memories , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[26]  F. Craik,et al.  Levels of Processing in Human Memory , 1979 .

[27]  B. S. Manjunath,et al.  Introduction to MPEG-7: Multimedia Content Description Interface , 2002 .

[28]  Anita L. Allen,et al.  Dredging Up the Past: Lifelogging, Memory and Surveillance , 2007 .

[29]  Alan F. Smeaton,et al.  The SenseCam as a tool for task observation , 2008 .

[30]  Kiyoharu Aizawa,et al.  Context-based video retrieval system for the life-log applications , 2003, MIR '03.

[31]  Noel E. O'Connor,et al.  The acetoolbox: low-level audiovisual feature extraction for retrieval and classification , 2005 .

[32]  Alan F. Smeaton,et al.  Validating the Detection of Everyday Concepts in Visual Lifelogs , 2008, SAMT.

[33]  Alan F. Smeaton,et al.  Using text search for personal photo collections with the MediAssist system , 2007, SAC '07.

[34]  Robert L. Greene,et al.  Human Memory: Paradigms and Paradoxes , 1992 .

[35]  H. L. Hardman,et al.  Generating multimedia presentations : It's all in the game , 2004 .

[36]  Javed A. Aslam,et al.  Relevance score normalization for metasearch , 2001, CIKM '01.

[37]  L. Bannon.  Forgetting as a feature, not a bug: the dualityof memory and implications for ubiquitous computing , 2006 .

[38]  B. N. Chatterji,et al.  Comparison of similarity metrics for texture image retrieval , 2003, TENCON 2003. Conference on Convergent Technologies for Asia-Pacific Region.

[39]  Geraldine Fitzpatrick,et al.  Supporting collaborative reflection with passive image capture , 2006 .

[40]  B. Boruff,et al.  Short-term memory capacity: magic number or magic spell? , 1986, Journal of experimental psychology. Learning, memory, and cognition.

[41]  Caroline Parker,et al.  An examination of the effects of a wearable display on informal face-to-face communication , 2006, CHI.

[42]  Paul Over,et al.  TRECVID 2003 - an overview , 2003 .

[43]  A. Baddeley,et al.  Context-dependent memory in two natural environments: on land and underwater. , 1975 .

[44]  Steve Mann,et al.  Wearable Computing: A First Step Toward Personal Imaging , 1997, Computer.

[45]  Kiyoharu Aizawa,et al.  Summarizing wearable video , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[46]  Alan F. Smeaton,et al.  Keyframe detection in visual lifelogs , 2008, PETRA '08.

[47]  A.F. Smeaton,et al.  Combining Face Detection and Novelty to Identify Important Events in a Visual Lifelog , 2008, 2008 IEEE 8th International Conference on Computer and Information Technology Workshops.

[48]  D. Kahneman,et al.  A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method , 2004, Science.

[49]  Jade Goldstein-Stewart,et al.  The use of MMR, diversity-based reranking for reordering documents and producing summaries , 1998, SIGIR '98.

[50]  Bettina Berendt,et al.  Tags are not metadata, but "just more content" - to some people , 2007, ICWSM.

[51]  Mohamed Batouche,et al.  Virtualized Real Object Integration and Manipulation in an Augmented Scene , 2005, CAIP.

[52]  Hayri Sever,et al.  Comparison of Normalization Techniques for Metasearch , 2002, ADVIS.

[53]  Alan F. Smeaton,et al.  Indexing, browsing, and searching of digital video , 2005, Annu. Rev. Inf. Sci. Technol..

[54]  Kiyoharu Aizawa,et al.  Capture and Efficient Retrieval of Life Log , 2004 .

[55]  Kiyoharu Aizawa,et al.  Practical experience recording and indexing of Life Log video , 2005, CARPE '05.

[56]  Alan F. Smeaton,et al.  Finding New News: Novelty Detection in Broadcast News , 2005, AIRS.

[57]  Joo-Hwee Lim,et al.  Scene Recognition with Camera Phones for Tourist Information Access , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[58]  Walter Bender,et al.  Next-Generation Personal Memory Aids , 2004 .

[59]  Alan F. Smeaton,et al.  Automatic Text Searching For Personal Photos , 2006, SAMT.

[60]  Gordon Bell,et al.  MyLifeBits: a personal database for everything , 2006, CACM.

[61]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[62]  Albrecht Schmidt,et al.  Recognizing context for annotating a live life recording , 2007, Personal and Ubiquitous Computing.

[63]  George A. Miller,et al.  Introduction to WordNet: An On-line Lexical Database , 1990 .

[64]  Mary Czerwinski,et al.  An Investigation of Memory for Daily Computing Events , 2002 .

[65]  Alan F. Smeaton,et al.  Organising a daily visual diary using multifeature clustering , 2007, Electronic Imaging.

[66]  Abigail Sellen,et al.  Do life-logging technologies support memory for the past?: an experimental study using sensecam , 2007, CHI.

[67]  Kiyoharu Aizawa,et al.  Novel Concept for Video Retrieval in Life Log Application , 2004, PCM.

[68]  A. Baddeley.  Your Memory: A User's Guide , 1982 .

[69]  Matt Jones,et al.  Autoethnography: a tool for practice and education , 2005, CHINZ '05.

[70]  Dan P. McAdams.  The Redemptive Self: Narrative Identity in America Today. , 2004 .

[71]  Kent Lyons,et al.  Capturing experiences anytime, anywhere , 2006, IEEE Pervasive Computing.

[72]  Alan F. Smeaton,et al.  An Examination of a Large Visual Lifelog , 2008, AIRS.

[73]  Noel E. O'Connor,et al.  Exploiting context information to aid landmark detection in SenseCam images , 2006 .

[74]  Sameer Singh,et al.  Novelty detection: a review - part 1: statistical approaches , 2003, Signal Process..

[75]  Andrew K. C. Wong,et al.  A new method for gray-level picture thresholding using the entropy of the histogram , 1985, Comput. Vis. Graph. Image Process..

[76]  Jane Greenberg,et al.  Memex Metadata ( M 2 ) for Reflective Learning , 2006 .

[77]  Sid Reich,et al.  Deja view camwear model 100 , 2004, CARPE'04.

[78]  Daniel P. W. Ellis,et al.  Accessing Minimal-Impact Personal Audio Archives , 2006, IEEE MultiMedia.

[79]  Ellen M. Voorhees,et al.  TREC: Continuing information retrieval's tradition of experimentation , 2007, CACM.

[80]  Martin A. Conway,et al.  Memory and the self , 2005 .

[81]  Kiyoharu Aizawa.  Emerging Issues for Multimedia Analysis and Applications , 2007, MCAM.

[82]  John R. Smith,et al.  On the detection of semantic concepts at TRECVID , 2004, MULTIMEDIA '04.

[83]  David R. Bull,et al.  Video Retrieval Using Global Features in Keyframes , 2002, TREC.

[84]  Gretchen Anderson,et al.  Why Consumers (Don't) Adopt Smart Wearable Electronics , 2008, IEEE Pervasive Computing.

[85]  Alex Pentland,et al.  InSense: Interest-Based Life Logging , 2006, IEEE MultiMedia.

[86]  Catherine C. Marshall,et al.  Keeping encountered information , 2006, CACM.

[87]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[88]  John A. King,et al.  Memory for events and their spatial context: models and experiments. , 2001, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[89]  Jane Greenberg,et al.  Augmenting memory for student learning: Designing a context-aware capture system for biology education , 2006, ASIST.

[90]  N. J. Slamecka,et al.  The Generation Effect: Delineation of a Phenomenon , 1978 .

[91]  Dean S. Messing,et al.  The MPEG-7 colour structure descriptor: image description using colour and local spatial information , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[92]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[93]  Keansub Lee,et al.  Minimal-impact audio-based personal archives , 2004, CARPE'04.

[94]  Noel E. O'Connor,et al.  Towards Hardware Acceleration of Neuroevolution for Multimedia Processing Applications on Mobile Devices , 2006, ICONIP.

[95]  Alan F. Smeaton,et al.  Multimodal Segmentation of Lifelog Data , 2007, RIAO.

[96]  Alan F. Smeaton,et al.  Aggregating multiple body sensors for analysis in sports , 2008 .

[97]  Paul Over,et al.  TRECVID: evaluating the effectiveness of information retrieval tasks on digital video , 2004, MULTIMEDIA '04.

[98]  Jim Gemmell,et al.  Telling Stories with Mylifebits , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[99]  Nigel Shadbolt,et al.  Lifelogging: Issues of Identity and Privacy with Memories for Life , 2008 .

[100]  Zongpeng Li,et al.  Youtube traffic characterization: a view from the edge , 2007, IMC '07.

[101]  Roelof van Zwol,et al.  Flickr: Who is Looking? , 2007, Web Intelligence.

[102]  Kiyoharu Aizawa,et al.  Practical life log video indexing based on content and context , 2006, Electronic Imaging.

[103]  Alan F. Smeaton,et al.  Using Graphics Processor Units (GPUs) for Automatic Video Structuring , 2007, Eighth International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS '07).

[104]  Ciar´an´o Conaire.  Dynamic Thresholding Methods , .

[105]  A. Smeaton,et al.  Combination of content analysis and context features for digital photograph retrieval. , 2005 .

[106]  Noel E. O'Connor,et al.  Mo Músaem Fíorúil: A Web-Based Search and Information Service for Museum Visitors , 2008, ICIAR.

[107]  Alan F. Smeaton,et al.  Combining image descriptors to effectively retrieve events from visual lifelogs , 2008, MIR '08.

[108]  John R. Anderson.  Language, Memory, and Thought , 1976 .

[109]  Hideaki Takeda,et al.  Ubiquitous Memories: a memory externalization system using physical objects , 2007, Personal and Ubiquitous Computing.

[110]  Andreas Paepcke,et al.  Time as essence for photo browsing through personal digital libraries , 2002, JCDL '02.

[111]  P. Schmitz,et al.  Inducing Ontology from Flickr Tags , 2006 .

[112]  D. Rubin.  Remembering our past : studies in autobiographical memory , 1996 .

[113]  James Fung,et al.  Designing EyeTap Digital Eyeglasses for Continuous Lifelong Capture and Sharing of Personal Experiences , 2005 .

[114]  D G Gadian,et al.  Dissociations in cognitive memory: the syndrome of developmental amnesia. , 2001, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[115]  John Adcock,et al.  Video summarization preserving dynamic content , 2007, TVS '07.

[116]  Ig-Jae Kim,et al.  PERSONE: personalized experience recoding and searching on networked environment , 2006, CARPE '06.

[117]  Daniel L. Schacter,et al.  Suppressing False Recognition in Younger and Older Adults: The Distinctiveness Heuristic ☆ ☆☆ ★ , 1999 .

[118]  P. E. Morris,et al.  Practical aspects of memory : current research and issues , 1988 .

[119]  Christian Plaunt,et al.  Subtopic structuring for full-length document access , 1993, SIGIR.

[120]  Bernard Mérialdo,et al.  Split-screen dynamically accelerated video summaries , 2007, TVS '07.

[121]  Qi Tian,et al.  Semantic Retrieval of Video , 2006 .

[122]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[123]  Frank Nack,et al.  You Must Remember This , 2005, IEEE Multim..

[124]  Bülent Sankur,et al.  Survey over image thresholding techniques and quantitative performance evaluation , 2004, J. Electronic Imaging.

[125]  Boon-Lock Yeo,et al.  Time-constrained clustering for segmentation of video into story units , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[126]  G. Bell,et al.  A digital life , 2007 .

[127]  Chris Baber,et al.  Defining and evaluating context for wearable computing , 2004, Int. J. Hum. Comput. Stud..

[128]  Shun'ichi Tano,et al.  Multimedia Informal Communication by Wearable Computer based on Real-World Context and Graffiti , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[129]  Gordon Bell,et al.  Passive capture and ensuing issues for a personal lifetime store , 2004, CARPE'04.

[130]  Darren Newtson,et al.  The objective basis of behavior units. , 1977 .

[131]  Alan F. Smeaton,et al.  Automatically Segmenting LifeLog Data into Events , 2008, 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services.

[132]  Joo-Hwee Lim,et al.  Object Identification and Retrieval from Efficient Image Matching: Snap2Tell with the STOIC Dataset , 2005, AIRS.

[133]  Wei-Hao Lin,et al.  Structuring continuous video recordings of everyday life using time-constrained clustering , 2006, Electronic Imaging.

[134]  Georgina Gaughan.  Novelty detection in video retrieval: finding new news in TV news stories , 2006 .

[135]  Michael Bukhin,et al.  WayMarkr: acquiring perspective through continuous documentation , 2006, MUM '06.

[136]  Jeffrey M. Zacks,et al.  Event understanding and memory in healthy aging and dementia of the Alzheimer type. , 2006, Psychology and aging.

[137]  Alan F. Smeaton,et al.  Using bluetooth and GPS metadata to measure event similarity in SenseCam Images , 2007 .

[138]  Deborah Estrin,et al.  Image browsing, processing, and clustering for participatory sensing: lessons from a DietSense prototype , 2007, EmNets '07.

[139]  Alan F. Smeaton,et al.  Investigating keyframe selection methods in the novel domain of passively captured visual lifelogs , 2008, CIVR '08.

[140]  Nasser Peyghambarian.  Thanks for the memory , 1995, Nature.

[141]  David Elsweiler,et al.  Supporting human memory in personal information management , 2008, SIGF.