Multimodal technologies for seniors: challenges and opportunities

This chapter discusses interactive technologies in the service of seniors. Adults over 65 form one of the largest and most rapidly growing user groups in the industrialized society. Interactive technologies have been steadily improving in their ability to provide practical support for these users and their daily challenges. These applications range from supporting low-literacy adults with mobile touch and spoken language interfaces [Munteanu et al. 2014], to interactive robotic fitness coaches for seniors [Gorer et al. 2016]. The goal of this chapter is to present an in-depth analysis of the needs of seniors, as well as to investigate how multimodal and multisensor technologies are used to address these needs. In Section 8.2 of this chapter, we structure the needs of older adults according to a hierarchy of human needs that emphasizes self-actualization [Maslow 1954]. Maslow's theoretical prism helps us to identify a broad spectrum of potential opportunities for researchers and practitioners who design multimodal-multisensor applications. We present a number of example applications in Section 8.3. These examples are illustrative of how interactive technologies, especially ones that leverage multiple modalities (e.g., speech, gestures, touch) and sensor data (e.g., location and accelerometers available on consumer phones) can provide a basis for meeting the needs of older adults. Section 8.4 briefly surveys future directions of multimodal-multisensor research that has potential for developing valuable new 320 Chapter 8 Multimodal Technologies for Seniors: Challenges and Opportunities assistive living technologies.1 Section 8.5 is a discussion of both design and implementation challenges, followed with conclusions in Section 8.6, and a list of supplementary Focus Questions to aid in further understanding of essential content. We hope that this broad overview will encourage both new and established researchers to explore how current or future multimodal and multisensor interfaces can better support one of the largest but most under-represented demographic groups.

[1]  Shyamal Patel,et al.  A review of wearable sensors and systems with application in rehabilitation , 2012, Journal of NeuroEngineering and Rehabilitation.

[2]  Rachel L. Franz,et al.  "My Hand Doesn't Listen to Me!": Adoption and Evaluation of a Communication Technology for the 'Oldest Old' , 2015, CHI.

[3]  Nadine Vigouroux,et al.  Touch screen user interfaces for older subjects: effect of the targets number and the two hands use , 2010 .

[4]  Jörn Hurtienne,et al.  Multimodal interaction: A suitable strategy for including older users? , 2010, Interact. Comput..

[5]  M. Nilsson,et al.  The Soft Extra Muscle system for improving the grasping capability in neurological rehabilitation , 2012, 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences.

[6]  Matthew Turk,et al.  Multimodal interaction: A review , 2014, Pattern Recognit. Lett..

[7]  S. Zwijsen,et al.  Ethics of using assistive technology in the care for community-dwelling elderly people: An overview of the literature , 2011, Aging & mental health.

[8]  Goldie Nejat,et al.  Recognizing Emotional Body Language Displayed by a Human-like Social Robot , 2014, International Journal of Social Robotics.

[9]  Steve Renals,et al.  Longitudinal study of ASR performance on ageing voices , 2008, INTERSPEECH.

[10]  Mukhtiar Memon,et al.  Ambient Assisted Living Healthcare Frameworks, Platforms, Standards, and Quality Attributes , 2014, Sensors.

[11]  Cem Ersoy,et al.  Wireless sensor networks for healthcare: A survey , 2010, Comput. Networks.

[12]  Babak A. Farshchian,et al.  The role of ICT in addressing the challenges of age-related falls: a research agenda based on a systematic mapping of the literature , 2015, Personal and Ubiquitous Computing.

[13]  Yvonne Rogers,et al.  Interaction Design: Beyond Human-Computer Interaction , 2002 .

[14]  F. Amaro,et al.  Coming of (Old) Age in the Digital Age: ICT Usage and Non-Usage among Older Adults , 2013 .

[15]  M. Claver,et al.  Perceived control in the lives of older adults: the influence of Langer and Rodin's work on gerontological theory, policy, and practice. , 2014, The Gerontologist.

[16]  E. Campo,et al.  Homecare monitoring system: A technical proposal for the safety of the elderly experimented in an Alzheimer's care unit , 2013 .

[17]  Joelle Pineau,et al.  Towards robotic assistants in nursing homes: Challenges and results , 2003, Robotics Auton. Syst..

[18]  Panos Markopoulos,et al.  Aurama: caregiver awareness for living independently with an augmented picture frame display , 2010, AI & SOCIETY.

[19]  Gerhard Fischer,et al.  Design, adoption, and assessment of a socio-technical environment supporting independence for persons with cognitive disabilities , 2008, CHI.

[20]  Alex Mihailidis,et al.  A Survey on Ambient-Assisted Living Tools for Older Adults , 2013, IEEE Journal of Biomedical and Health Informatics.

[21]  A. Maslow A Theory of Human Motivation , 1943 .

[22]  Miguel Sales Dias,et al.  Towards Elderly Social Integration using a Multimodal Human-computer Interface , 2016 .

[23]  J. Cameron,et al.  The relationship between collective self-esteem and self-construal in Japan and Canada. , 1999, The Journal of social psychology.

[24]  Chandra M. Harrison Low-vision reading aids: reading as a pleasurable experience , 2004, Personal and Ubiquitous Computing.

[25]  Chris Todd,et al.  Older adults' perceptions of technologies aimed at falls prevention, detection or monitoring: A systematic review , 2014, Int. J. Medical Informatics.

[26]  Eric Campo,et al.  A review of smart homes - Present state and future challenges , 2008, Comput. Methods Programs Biomed..

[27]  A. van Berlo Smart home technology: Have older people paved the way? , 2002 .

[28]  Mark Perry,et al.  Multimodal and ubiquitous computing systems: supporting independent-living older users , 2004, IEEE Transactions on Information Technology in Biomedicine.

[29]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[30]  James Baker,et al.  A historical perspective of speech recognition , 2014, CACM.

[31]  Sharon L. Oviatt,et al.  Toward a theory of organized multimodal integration patterns during human-computer interaction , 2003, ICMI '03.

[32]  Goldie Nejat,et al.  Brian 2.1: A socially assistive robot for the elderly and cognitively impaired , 2013, IEEE Robotics & Automation Magazine.

[33]  Mark A. Neerincx,et al.  Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors , 2010, Int. J. Hum. Comput. Stud..

[34]  Yi-Ru Regina Chen,et al.  The Effect of Information Communication Technology Interventions on Reducing Social Isolation in the Elderly: A Systematic Review , 2016, Journal of medical Internet research.

[35]  Kent L. Norman,et al.  GEQ (Game Engagement/Experience Questionnaire): A Review of Two Papers , 2013, Interact. Comput..

[36]  Maja J. Mataric,et al.  Using Socially Assistive Human–Robot Interaction to Motivate Physical Exercise for Older Adults , 2012, Proceedings of the IEEE.

[37]  P. Kannus,et al.  Prevention of falls and consequent injuries in elderly people , 2005, The Lancet.

[38]  Maria Klara Wolters,et al.  User-centred multimodal reminders for assistive living , 2011, CHI.

[39]  Carman Neustaedter,et al.  Intimacy in long-distance relationships over video chat , 2012, CHI.

[40]  Suranga Nanayakkara,et al.  FingerReader: A Wearable Device to Explore Printed Text on the Go , 2015, CHI.

[41]  P. Dario,et al.  Design strategies to improve patient motivation during robot-aided rehabilitation , 2007, Journal of NeuroEngineering and Rehabilitation.

[42]  K. Covinsky,et al.  Loneliness in older persons: a predictor of functional decline and death. , 2012, Archives of internal medicine.

[43]  Stefano Ramat,et al.  A Wearable and Modular Inertial Unit for Measuring Limb Movements and Balance Control Abilities , 2016, IEEE Sensors Journal.

[44]  Jenny Waycott,et al.  Captioned photographs in psychosocial aged care: relationship building and boundary work , 2014, CHI.

[45]  Euan Freeman,et al.  Multimodal feedback in HCI: haptics, non-speech audio, and their applications , 2017, The Handbook of Multimodal-Multisensor Interfaces, Volume 1.

[46]  Stephen Lindsay,et al.  Questionable concepts: critique as resource for designing with eighty somethings , 2012, CHI.

[47]  M. Tinetti,et al.  Potential pitfalls of disease-specific guidelines for patients with multiple conditions. , 2004, The New England journal of medicine.

[48]  Rana El Kaliouby,et al.  On the Future of Personal Assistants , 2016, CHI Extended Abstracts.

[49]  Gary S. Dell,et al.  Language production and comprehension , 2010 .

[50]  Sharon L. Oviatt,et al.  Breaking the Robustness Barrier: Recent Progress on the Design of Robust Multimodal Systems , 2002, Adv. Comput..

[51]  Benoit M. Macq,et al.  User-Centered Design and Fast Prototyping of an Ambient Assisted Living System for Elderly People , 2009, 2009 Sixth International Conference on Information Technology: New Generations.

[52]  António J. S. Teixeira,et al.  Multilingual Speech Recognition for the Elderly: The AALFred Personal Life Assistant , 2015, DSAI.

[53]  Eva Siegenthaler,et al.  Improving the Usability of E-Book Readers , 2010 .

[54]  David V. Keyson,et al.  SnowGlobe: the development of a prototype awareness system for longitudinal field studies , 2010, Conference on Designing Interactive Systems.

[55]  Ronald Baecker,et al.  An accessible, large-print, listening and talking e-book to support families reading together , 2013, MobileHCI '13.

[56]  Alexander H. Waibel,et al.  Enabling Multimodal Human–Robot Interaction for the Karlsruhe Humanoid Robot , 2007, IEEE Transactions on Robotics.

[57]  Sharon L. Oviatt,et al.  Advances in Robust Multimodal Interface Design , 2003, IEEE Computer Graphics and Applications.

[58]  Davide Carneiro,et al.  A multi-modal approach for activity classification and fall detection , 2014, Int. J. Syst. Sci..

[59]  Lynne Baillie,et al.  Senior designers: empowering seniors to design enjoyable falls rehabilitation tools , 2012, CHI.

[60]  David J. Reinkensmeyer,et al.  Gesture Therapy: A Vision-Based System for Arm Rehabilitation after Stroke , 2008, BIOSTEC.

[61]  António J. S. Teixeira,et al.  Speech-centric Multimodal Interaction for Easy-to-access Online Services - A Personal Life Assistant for the Elderly , 2013, DSAI.

[62]  Lu Yang,et al.  A Survey on Media Interaction in Social Robotics , 2015, PCM.

[63]  Cosmin Munteanu,et al.  An Information-Centric Framework for Mobile Collaboration Between Seniors and Caregivers that Balances Independence, Privacy, and Social Connectedness , 2016, HCI.

[64]  Julie Maitland,et al.  Hidden in plain sight: low-literacy adults in a developed country overcoming social and educational challenges through mobile learning support tools , 2014, Personal and Ubiquitous Computing.

[65]  John F. Canny,et al.  Balancing Accuracy and Fun: Designing Camera Based Mobile Games for Implicit Heart Rate Monitoring , 2015, CHI.

[66]  Ling Shao,et al.  Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Karon E. MacLean,et al.  Multisensory haptic interactions: understanding the sense and designing for it , 2017, The Handbook of Multimodal-Multisensor Interfaces, Volume 1.

[68]  Martina Ziefle,et al.  Visual and Auditory Interfaces of Advanced Driver Assistant Systems for Older Drivers , 2008, ICCHP.

[69]  Stefan Kopp,et al.  Virtual Agents as Daily Assistants for Elderly or Cognitively Impaired People - Studies on Acceptance and Interaction Feasibility , 2013, IVA.

[70]  Carlos Duarte,et al.  The Contribution of Multimodal Adaptation Techniques to the GUIDE Interface , 2011, HCI.

[71]  Alex Mihailidis,et al.  The design of intelligent in-home assistive technologies: Assessing the needs of older adults with dementia and their caregivers , 2011 .

[72]  Patrick Langdon,et al.  Developing accessible TV applications , 2011, ASSETS '11.

[73]  Marcel Heerink,et al.  Assessing acceptance of assistive social robots by aging adults , 2010 .

[74]  Eric Dishman,et al.  Inventing wellness systems for aging in place , 2004, Computer.

[75]  Shwetak N. Patel,et al.  Maslow’s Hierarchy of Human Needs and the Adoption of Health-Related Technologies for Older Adults , 2012 .

[76]  Ayanna K. Thomas,et al.  Theoretical Perspectives on Cognitive Aging , 2019, Handbook of Medical Neuropsychology.

[77]  Hiroshi Ishii,et al.  People in books: using a FlashCam to become part of an interactive book for connected reading , 2012, CSCW.

[78]  Julie Maitland,et al.  "Showing off" your mobile device: adult literacy learning in the classroom and beyond , 2011, Mobile HCI.

[79]  Antonio Teixeira,et al.  Multimodal and adaptable medication assistant for the elderly: A prototype for interaction and usability in smartphones , 2013, 2013 8th Iberian Conference on Information Systems and Technologies (CISTI).

[80]  Ahmed Nait Aicha,et al.  Hipperbox: a portable toolbox for rehabilitation from a hip surgery using sensors , 2016, PervasiveHealth.

[81]  Yvonne Rogers,et al.  Fat Finger Worries: How Older and Younger Users Physically Interact with PDAs , 2005, INTERACT.

[82]  Gregory D. Abowd,et al.  The Aware Home: A living laboratory for technologies for successful aging , 2002 .

[83]  G. Demiris,et al.  Technologies for an Aging Society: A Systematic Review of “Smart Home” Applications , 2008, Yearbook of Medical Informatics.

[84]  Albert A. Rizzo,et al.  User-State Sensing for Virtual Health Agents and TeleHealth Applications , 2013, MMVR.

[85]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[86]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[87]  Albert Ali Salah,et al.  A Robotic Fitness Coach for the Elderly , 2013, AmI.

[88]  Mark D. Dunlop,et al.  A Participatory Design and Formal Study Investigation into Mobile Text Entry for Older Adults , 2016, Int. J. Mob. Hum. Comput. Interact..

[89]  D. Feil-Seifer,et al.  Defining socially assistive robotics , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[90]  António J. S. Teixeira,et al.  Elderly Centered Design for Interaction - The Case of the S4S Medication Assistant , 2013, DSAI.

[91]  Sergio F. Ochoa,et al.  Introducing Ambient Assisted Living Technology at the Home of the Elderly: Challenges and Lessons Learned , 2015, IWAAL.

[92]  Audie A Atienza,et al.  Mobile health technology evaluation: the mHealth evidence workshop. , 2013, American journal of preventive medicine.

[93]  Paolo Dario,et al.  Robot-Era Project: Preliminary Results on the System Usability , 2015, HCI.

[94]  Daniel Sonntag,et al.  Robot Companions and Smartpens for Improved Social Communication of Dementia Patients , 2015, IUI Companion.

[95]  C. Becker,et al.  Smartphone-based solutions for fall detection and prevention: the FARSEEING approach , 2012, Zeitschrift für Gerontologie und Geriatrie.

[96]  A. Maslow Motivation and Personality , 1954 .

[97]  Diane J. Cook,et al.  Behavior Analysis for Elderly , 2015, HBU.

[98]  Ben J. A. Kröse,et al.  Assessing Acceptance of Assistive Social Agent Technology by Older Adults: the Almere Model , 2010, Int. J. Soc. Robotics.

[99]  Adriana Tapus,et al.  Socially Assistive Robots: The Link between Personality, Empathy, Physiological Signals, and Task Performance , 2008, AAAI Spring Symposium: Emotion, Personality, and Social Behavior.

[100]  Julie A. Kientz,et al.  Eyes-free yoga: an exergame using depth cameras for blind & low vision exercise , 2013, ASSETS.

[101]  Juan Fasola,et al.  A socially assistive robot exercise coach for the elderly , 2013, J. Hum. Robot Interact..

[102]  Yvonne Barnard,et al.  Learning to use new technologies by older adults: Perceived difficulties, experimentation behaviour and usability , 2013, Comput. Hum. Behav..

[103]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[104]  Karyn Moffatt,et al.  Older-adult HCI: why should we care? , 2013, INTR.

[105]  Per Ola Kristensson,et al.  None of a CHInd: relationship counselling for HCI and speech technology , 2014, CHI Extended Abstracts.

[106]  Bastian Pfleging,et al.  SpeeT: A Multimodal Interaction Style Combining Speech and Touch Interaction in Automotive Environments , 2011 .

[107]  Mary Zajicek,et al.  Multimodality and interactional differences in older adults , 2003, Universal Access in the Information Society.

[108]  Cui Jian,et al.  Modality Preference in Multimodal Interaction for Elderly Persons , 2013, BIOSTEC.

[109]  Kanya Tanaka,et al.  Meal-Assistance Robot Using Ultrasonic Motor with Eye Interface , 2014, Int. J. Autom. Technol..

[110]  Goldie Nejat,et al.  Casper: An Assistive Kitchen Robot to Promote Aging in Place , 2014 .

[111]  D. Volkert,et al.  Nutritional status according to the Mini Nutritional Assessment (MNA®) and frailty in community dwelling older persons: A close relationship , 2013, The journal of nutrition, health & aging.

[112]  Timothy W. Bickmore,et al.  Acceptance and usability of a relational agent interface by urban older adults , 2005, CHI Extended Abstracts.

[113]  Frédéric Lerasle,et al.  A multi-modal perception based assistive robotic system for the elderly , 2016, Comput. Vis. Image Underst..

[114]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[115]  Rachel L. Franz,et al.  Time to Retire Old Methodologies? Reflecting on Conducting Usability Evaluations with Older Adults , 2015, MobileHCI Adjunct.

[116]  Takanori Shibata,et al.  Psychological and Social Effects of One Year Robot Assisted Activity on Elderly People at a Health Service Facility for the Aged , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[117]  Sharon L. Oviatt,et al.  Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences , 2003, ICMI '03.

[118]  Yi-Ying Lin,et al.  BlindNavi: A Navigation App for the Visually Impaired Smartphone User , 2015, CHI Extended Abstracts.

[119]  Alex Mihailidis,et al.  Towards a single sensor passive solution for automated fall detection , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[120]  Albert Ali Salah,et al.  An autonomous robotic exercise tutor for elderly people , 2017, Auton. Robots.

[121]  Chu Kiong Loo,et al.  Extensive assessment and evaluation methodologies on assistive social robots for modelling human-robot interaction - A review , 2015, Inf. Sci..

[122]  Albert Ali Salah,et al.  Coupling Fall Detection and Tracking in Omnidirectional Cameras , 2014, HBU.