Interactive Question-Posing System for Robot-Assisted Reminiscence From Personal Photographs

Reminiscence is a lifelong activity that happens throughout our lifespan. While memories can serve as topics in people’s everyday conversations, recalling the past can also help us build self-esteem and increase our level of happiness. In this paper, we aim to develop a robot companion that helps people to recollect their memories from personal photographs. We focus on how a robot can associate concepts relevant to the content in the photographs and evoke people’s memories by asking questions that are both relatable and engaging. To understand the content of a picture, we applied deep learning techniques in order to recognize events, objects, and scenes in it. Then, these observations and any user utterances are considered in a Markov random field (MRF)-based algorithm that contains common sense knowledge of a number of events, with loopy belief propagation being used to infer possible associated concepts and topics. Afterward, the robot poses appropriate questions about the selected topics, guiding the user to reminisce through conversation. Our results show that the proposed system can pose related and appropriate questions to interact with the user, and has the potential guide the user to recall the past in an organized way.

[1]  Mirko Gelsomini,et al.  Telling Stories to Robots: The Effect of Backchanneling on a Child's Storytelling * , 2017, 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI.

[2]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[3]  Lun-Wei Ku,et al.  Sensing Emotions in Text Messages: An Application and Deployment Study of EmotionPush , 2016, COLING.

[4]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[5]  Gregory M. P. O'Hare,et al.  Social interaction between robots, avatars & humans , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[6]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[7]  Hilbert J. Kappen,et al.  Sufficient Conditions for Convergence of Loopy Belief Propagation , 2005, UAI.

[8]  Janet L. Kolodner,et al.  Reconstructive Memory: A Computer Model , 1983, Cogn. Sci..

[9]  Ana Paiva,et al.  MAY: My Memories Are Yours , 2010, IVA.

[10]  Catherine Havasi,et al.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge , 2016, AAAI.

[11]  Kristiina Jokinen,et al.  Modelling User Experience in Human-Robot Interactions , 2014, MA3HMI@INTERSPEECH.

[12]  Norman Alm,et al.  A Communication Support System for Older People with Dementia , 2007, Computer.

[13]  Yao Ching-Teng,et al.  Effects of structured group reminiscence therapy on the life satisfaction of institutionalized older adults in Taiwan , 2018, Social work in health care.

[14]  F. Bartlett,et al.  Remembering: A Study in Experimental and Social Psychology , 1932 .

[15]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Nicu Sebe,et al.  Exploitation of time constraints for (sub-)event recognition , 2011, J-MRE '11.

[17]  Reza Fuad Rachmadi,et al.  Spatial Pyramid Convolutional Neural Network for Social Event Detection in Static Image , 2016, ArXiv.

[18]  Jens Lehmann,et al.  DBpedia: A Nucleus for a Web of Open Data , 2007, ISWC/ASWC.

[19]  F. Bryant,et al.  Using the Past to Enhance the Present: Boosting Happiness Through Positive Reminiscence , 2005 .

[20]  M Orrell,et al.  Reminiscence therapy for dementia (Review) , 2009 .

[21]  Hiroshi Ishiguro,et al.  Generation of nodding, head tilting and eye gazing for human-robot dialogue interaction , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[22]  Francesco G. B. De Natale,et al.  USED: a large-scale social event detection dataset , 2016, MMSys.

[23]  Chunhua Shen,et al.  Explicit Knowledge-based Reasoning for Visual Question Answering , 2015, IJCAI.

[24]  Richard C. Atkinson,et al.  Human Memory: A Proposed System and its Control Processes , 1968, Psychology of Learning and Motivation.

[25]  Dan Cosley,et al.  Pensieve: supporting everyday reminiscence , 2010, CHI.

[26]  Jianfeng Gao,et al.  Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation , 2017, IJCNLP.

[27]  N. S. Johnson,et al.  Remembrance of things parsed: Story structure and recall , 1977, Cognitive Psychology.

[28]  Jianfeng Gao,et al.  Deep Reinforcement Learning for Dialogue Generation , 2016, EMNLP.

[29]  John B. Black,et al.  Knowledge structures in the organization and retrieval of autobiographical memories , 1985, Cognitive Psychology.

[30]  Yong Yu,et al.  Searching Questions by Identifying Question Topic and Question Focus , 2008, ACL.

[31]  Björn W. Schuller,et al.  SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives , 2016, COLING.

[32]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[33]  Alan Ritter,et al.  Adversarial Learning for Neural Dialogue Generation , 2017, EMNLP.

[34]  Bolei Zhou,et al.  Places: A 10 Million Image Database for Scene Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Margaret Mitchell,et al.  VQA: Visual Question Answering , 2015, International Journal of Computer Vision.

[36]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[37]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[38]  Jeffrey M. Zacks,et al.  Human brain activity time-locked to perceptual event boundaries , 2001, Nature Neuroscience.

[39]  Michael F. McTear,et al.  MemoryLane: reminiscence for older adults , 2009 .

[40]  Chitta Baral,et al.  Image Understanding using vision and reasoning through Scene Description Graph , 2018, Comput. Vis. Image Underst..

[41]  Fei-Fei Li,et al.  Deep visual-semantic alignments for generating image descriptions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Robin R. Murphy,et al.  Survey of metrics for human-robot interaction , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[43]  Margaret Mitchell,et al.  Generating Natural Questions About an Image , 2016, ACL.

[44]  Yiannis Kompatsiaris,et al.  Social Event Detection at MediaEval 2012: Challenges, Dataset and Evaluation , 2012, MediaEval.

[45]  Sheetal Rakangor,et al.  Literature Review of Automatic Question Generation Systems , 2015 .