Smart Assistive Architecture for the Integration of IoT Devices, Robotic Systems, and Multimodal Interfaces in Healthcare Environments

This paper presents a new architecture that integrates Internet of Things (IoT) devices, service robots, and users in a smart assistive environment. A new intuitive and multimodal interaction system supporting people with disabilities and bedbound patients is presented. This interaction system allows the user to control service robots and devices inside the room in five different ways: touch control, eye control, gesture control, voice control, and augmented reality control. The interaction system is comprised of an assistive robotic arm holding a tablet PC. The robotic arm can place the tablet PC in front of the user. A demonstration of the developed technology, a prototype of a smart room equipped with home automation devices, and the robotic assistive arm are presented. The results obtained from the use of the various interfaces and technologies are presented in the article. The results include user preference with regard to eye-base control (performing clicks, and using winks or gaze) and the use of mobile phones over augmented reality glasses, among others.

[1]  P. Gondim,et al.  IoMT Platform for Pervasive Healthcare Data Aggregation, Processing, and Sharing Based on OneM2M and OpenEHR , 2019, Sensors.

[2]  Kalpana Lamb,et al.  Hand gesture recognition based bed position control for disabled patients , 2016, 2016 Conference on Advances in Signal Processing (CASP).

[3]  Kenro Yajima,et al.  Robot Arm Utilized Having Meal Support System Based on Computer Input by Human Eyes Only , 2011 .

[4]  Luciano Gamberini,et al.  Internet of Things for Elderly and Fragile People , 2020, ArXiv.

[5]  Asif Shahriyar Sushmit,et al.  Design of a gesture controlled robotic gripper arm using neural networks , 2017, 2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI).

[6]  Nak Young Chong,et al.  An Integrated Approach to Human-Robot-Smart Environment Interaction Interface for Ambient Assisted Living , 2018, 2018 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO).

[7]  Ransalu Senanayake,et al.  A robust vision-based hand gesture recognition system for appliance control in smart homes , 2012, 2012 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2012).

[8]  Feng Zhao,et al.  ENRICHME: Perception and Interaction of an Assistive Robot for the Elderly at Home , 2020, Int. J. Soc. Robotics.

[9]  Krzysztof Czuszynski,et al.  Towards Contactless, Hand Gestures-Based Control of Devices , 2018, IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society.

[10]  J.D. Enderle,et al.  The assistive robotic arm , 2007, 2007 IEEE 33rd Annual Northeast Bioengineering Conference.

[11]  Giancarlo Mauri,et al.  System of Nudge Theory-Based ICT Applications for Older Citizens: The SENIOR Project , 2019, MindCare.

[12]  Hyunsik Ahn,et al.  A multimodal ubiquitous interface system using smart phone for human-robot interaction , 2011, 2011 8th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[13]  Dorin Moldovan,et al.  Smart Environments and Social Robots for Age-Friendly Integrated Care Services , 2020, International journal of environmental research and public health.

[14]  C. Evinger,et al.  Asymmetry of blinking. , 2006, Investigative ophthalmology & visual science.

[15]  Hideyuki Uehara,et al.  A mobile robotic arm for people with severe disabilities , 2010, 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.

[16]  Jacopo M. Araujo,et al.  Exploring Eye-Gaze Wheelchair Control , 2020, ETRA Adjunct.

[17]  Paolo Dario,et al.  On the design, development and experimentation of the ASTRO assistive robot integrated in smart environments , 2013, 2013 IEEE International Conference on Robotics and Automation.

[18]  Yongho Seo,et al.  Controlling Mobile Robot Using IMU and EMG Sensor-Based Gesture Recognition , 2014, 2014 Ninth International Conference on Broadband and Wireless Computing, Communication and Applications.

[19]  Vladimir Trajkovik,et al.  Smart home environment aimed for people with physical disabilities , 2016, 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing (ICCP).

[20]  Progress Mtshali,et al.  A Smart Home Appliance Control System for Physically Disabled People , 2019, 2019 Conference on Information Communications Technology and Society (ICTAS).

[21]  Wan Ling Chang,et al.  Potential use of robots in Taiwanese nursing homes , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[22]  Didier Stricker,et al.  Continuous natural user interface: Reducing the gap between real and digital world , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[23]  Alessandro Saffiotti,et al.  PEIS Ecology: integrating robots into smart environments , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[24]  Marcos Sacristán Herrero,et al.  Robohealth: smart room interface for hospital patients , 2018 .

[25]  Cheonshu Park,et al.  A study on service robot system for elder care , 2012, 2012 9th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[26]  Hyung-Soon Park,et al.  Development of a Robotic Arm for Handicapped People: A Task-Oriented Design Approach , 2001, Auton. Robots.

[27]  Ivan Drozd,et al.  Multimodal interface for smart home , 2016, 2016 International Symposium ELMAR.

[28]  Seiichiro Matsumoto,et al.  Echonet: A Home Network Standard , 2010, IEEE Pervasive Computing.

[29]  Takanori Shibata,et al.  Effects of robot-assisted activity for elderly people and nurses at a day service center , 2004, Proceedings of the IEEE.

[30]  William W. Abbott,et al.  3D gaze cursor: Continuous calibration and end-point grasp control of robotic actuators , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Carlos Carrascosa,et al.  An IoT and Fog Computing-Based Monitoring System for Cardiovascular Patients with Automatic ECG Classification Using Deep Neural Networks , 2020, Sensors.

[32]  Adel M. Alimi,et al.  Smart Tablet Monitoring by a Real-Time Head Movement and Eye Gestures Recognition System , 2014, 2014 International Conference on Future Internet of Things and Cloud.

[33]  Veton Kepuska,et al.  Next-generation of virtual personal assistants (Microsoft Cortana, Apple Siri, Amazon Alexa and Google Home) , 2018, 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC).

[34]  A. Aldo Faisal,et al.  Towards free 3D end-point control for robotic-assisted human reaching using binocular eye tracking , 2017, 2017 International Conference on Rehabilitation Robotics (ICORR).

[35]  Stefano Stramigioli,et al.  Lending a helping hand: toward novel assistive robotic arms , 2013, IEEE Robotics & Automation Magazine.

[36]  Ömer Nezih Gerek,et al.  A Low-Computational Approach on Gaze Estimation With Eye Touch System , 2014, IEEE Transactions on Cybernetics.

[37]  Alessandro Saffiotti,et al.  Integrating Robots into Smart Environments , 2006 .

[38]  Laura Montanini,et al.  User Interfaces in Smart Assistive Environments: Requirements, Devices, Applications , 2016 .

[39]  Stefano Tomasin,et al.  Requirements and Enablers of Advanced Healthcare Services over Future Cellular Systems , 2019, IEEE Communications Magazine.

[40]  Fakhri Karray,et al.  Human Machine Interaction Platform for Home Care Support System , 2020, 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[41]  Z. Zenn Bien,et al.  A Steward Robot for Human-Friendly Human-Machine Interaction in a Smart House Environment , 2008, IEEE Transactions on Automation Science and Engineering.

[42]  Thar Baker,et al.  Towards fog driven IoT healthcare: challenges and framework of fog computing in healthcare , 2018, ICFNDS.

[43]  Roope Raisamo,et al.  Multimodal human-computer interaction: a constructive and empirical study , 1999 .

[44]  Vladimir Pavlovic,et al.  Toward multimodal human-computer interface , 1998, Proc. IEEE.

[45]  Hicham G. Elmongui,et al.  Painting with light: Gesture based light control in architectural settings , 2017, 2017 IEEE Symposium on 3D User Interfaces (3DUI).

[46]  A. Aldo Faisal,et al.  “Wink to grasp” — comparing eye, voice & EMG gesture control of grasp with soft-robotic gloves , 2017, 2017 International Conference on Rehabilitation Robotics (ICORR).

[47]  Xiangke Wang,et al.  ROS-based UAV control using hand gesture recognition , 2017, 2017 29th Chinese Control And Decision Conference (CCDC).