Artificial Vision Algorithms for Socially Assistive Robot Applications: A Review of the Literature

Today, computer vision algorithms are very important for different fields and applications, such as closed-circuit television security, health status monitoring, and recognizing a specific person or object and robotics. Regarding this topic, the present paper deals with a recent review of the literature on computer vision algorithms (recognition and tracking of faces, bodies, and objects) oriented towards socially assistive robot applications. The performance, frames per second (FPS) processing speed, and hardware implemented to run the algorithms are highlighted by comparing the available solutions. Moreover, this paper provides general information for researchers interested in knowing which vision algorithms are available, enabling them to select the one that is most suitable to include in their robotic system applications.

[1]  Nitish V. Thakor,et al.  Demonstration of a Semi-Autonomous Hybrid Brain–Machine Interface Using Human Intracranial EEG, Eye Tracking, and Computer Vision to Control a Robotic Upper Limb Prosthetic , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[2]  Francisco J. Perales López,et al.  Using a Social Robot to Evaluate Facial Expressions in the Wild , 2020, Sensors.

[3]  Mark Elshaw,et al.  Deep Learning for Real Time Facial Expression Recognition in Social Robots , 2018, ICONIP.

[4]  Fillia Makedon,et al.  Multimodal approach for cognitive task performance prediction from body postures, facial expressions and EEG signal , 2018, MCPMD@ICMI.

[5]  Alexandru Sorici,et al.  User Detection, Tracking and Recognition in Robot Assistive Care Scenarios , 2018, TAROS.

[6]  Dimitrios Tzovaras,et al.  Robust Human Pose Tracking For Realistic Service Robot Applications , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[7]  Moustapha El Hassan,et al.  Enhanced Face Detection Based on Haar-Like and MB-LBP Features , 2019 .

[8]  Ji-Hoon Jeong,et al.  Assistive Robotic Arm Control based on Brain-Machine Interface with Vision Guidance using Convolution Neural Network , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[9]  Theint Theint Htwe,et al.  Vision-Based Autonomous Human Tracking Mobile Robot , 2017 .

[10]  Mohamed Tahir,et al.  Face-recognition based security robot incorporating omnidirectional vision , 2015 .

[11]  Yuan-Fu Liao,et al.  User identification and prioritization control of service robot teleoperation , 2020, 2020 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom).

[12]  Kimitoshi Yamazaki,et al.  Recognition and manipulation integration for a daily assistive robot working on kitchen environments , 2010, 2010 IEEE International Conference on Robotics and Biomimetics.

[13]  Feng Zhao,et al.  ENRICHME: Perception and Interaction of an Assistive Robot for the Elderly at Home , 2020, Int. J. Soc. Robotics.

[14]  Cristian Sminchisescu,et al.  3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children with Autism , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Maria Kyrarini,et al.  RGB-D Camera based 3D Human Mouth Detection and Tracking Towards Robotic Feeding Assistance , 2018, PETRA.

[16]  Alessandro Tognetti,et al.  A Multimodal Perception Framework for Users Emotional State Assessment in Social Robotics , 2017, Future Internet.

[17]  Ashwin P. Dani,et al.  Intention Inference for Human-Robot Collaboration in Assistive Robotics , 2017 .

[18]  Sudipta Ghosh,et al.  A Real-Time Face Motion Based Approach towards Modeling Socially Assistive Wireless Robot Control with Voice Recognition , 2015 .

[19]  Rory A. Cooper,et al.  Three Dimensional Computer Vision-Based Alternative Control Method For Assistive Robotic Manipulator , 2016, ICRA 2016.

[20]  Nak Young Chong,et al.  Learning human behavior for emotional body expression in socially assistive robotics , 2017, 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[21]  C. Valadão,et al.  Development of a Socially Assistive Robot Controlled by Emotions Based on Heartbeats and Facial Temperature of Children with Autistic Spectrum Disorder , 2020 .

[22]  Samuel Murray,et al.  Real-Time Multiple Object Tracking - A Study on the Importance of Speed , 2017, ArXiv.

[23]  Jean Massardi,et al.  PARC: A Plan and Activity Recognition Component for Assistive Robots , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Kimitoshi Yamazaki,et al.  Estimating Door Shape and Manipulation Model for Daily Assistive Robots Based on the Integration of Visual and Touch Information , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[25]  José Carlos Castillo,et al.  Social Robotics in Therapy of Apraxia of Speech , 2018, Journal of healthcare engineering.

[26]  Goldie Nejat,et al.  Determining the affective body language of older adults during socially assistive HRI , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  Jorge Luis Victória Barbosa,et al.  Assistive Robotics: Adaptive Multimodal Interaction Improving People with Communication Disorders , 2016 .

[28]  Miguel Cazorla,et al.  A Socially Assistive Robot for Elderly Exercise Promotion , 2019, IEEE Access.

[29]  Basilio Sierra,et al.  Shedding Light on People Action Recognition in Social Robotics by Means of Common Spatial Patterns , 2020, Sensors.

[30]  Juan Fasola,et al.  A socially assistive robot exercise coach for the elderly , 2013, J. Hum. Robot Interact..

[31]  Stefanos Zafeiriou,et al.  A Comprehensive Performance Evaluation of Deformable Face Tracking “In-the-Wild” , 2016, International Journal of Computer Vision.

[32]  Petros Maragos,et al.  Multimodal human action recognition in assistive human-robot interaction , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[33]  Robert Platt,et al.  Open world assistive grasping using laser selection , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Naoyuki Kubota,et al.  A Lightweight Neural-Net with Assistive Mobile Robot for Human Fall Detection System , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[35]  Haitao Wu,et al.  Human activity recognition based on the combined SVM&HMM , 2014, 2014 IEEE International Conference on Information and Automation (ICIA).

[36]  Ali Meghdari,et al.  Spontaneous Human-Robot Emotional Interaction Through Facial Expressions , 2016, ICSR.

[37]  Ayanna M. Howard,et al.  Identifying Engagement from Joint Kinematics Data for Robot Therapy Prompt Interventions for Children with Autism Spectrum Disorder , 2016, ICSR.

[38]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[39]  Taskin Padir,et al.  A cyber physical system testbed for assistive robotics technologies in the home , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[40]  Juan Fang,et al.  Vehicle-Mounted with Tracked Robotic System Based on the Kinect , 2019, 2019 2nd World Conference on Mechanical Engineering and Intelligent Manufacturing (WCMEIM).

[41]  Ahmad Lotfi,et al.  Human Activities Transfer Learning for Assistive Robotics , 2017, UKCI.

[42]  Christophe Leroux,et al.  ARMEN: Assistive robotics to maintain elderly people in natural environment , 2013 .

[43]  Ognjen Rudovic,et al.  Toward Personalized Affect-Aware Socially Assistive Robot Tutors in Long-Term Interventions for Children with Autism , 2021, ArXiv.

[44]  Gi Hyun Lim,et al.  Towards lifelong assistive robotics: A tight coupling between object perception and manipulation , 2018, Neurocomputing.

[45]  Antonis A. Argyros,et al.  Hobbit: Providing Fall Detection and Prevention for the Elderly in the Real World , 2018, J. Robotics.

[46]  Antonio Frisoli,et al.  A Robust Real-Time 3D Tracking Approach for Assisted Object Grasping , 2014, EuroHaptics.

[47]  Albert Ali Salah,et al.  An autonomous robotic exercise tutor for elderly people , 2017, Auton. Robots.

[48]  Berat A. Erol,et al.  Improved Deep Neural Network Object Tracking System for Applications in Home Robotics , 2018 .

[49]  Horst-Michael Groß,et al.  ROREAS: robot coach for walking and orientation training in clinical post-stroke rehabilitation—prototype implementation and evaluation in field trials , 2017, Auton. Robots.

[50]  Zhe Zhang,et al.  Human Body Pose Interpretation and Classification for Social Human-Robot Interaction , 2011, Int. J. Soc. Robotics.

[51]  Ya-Huei Wu,et al.  The Attitudes and Perceptions of Older Adults With Mild Cognitive Impairment Toward an Assistive Robot , 2016, Journal of applied gerontology : the official journal of the Southern Gerontological Society.

[52]  José Manuel Ferrández,et al.  Real-Time Emotional Recognition for Sociable Robotics Based on Deep Neural Networks Ensemble , 2019, IWINAC.

[53]  Ahmad Lotfi,et al.  Human activity learning for assistive robotics using a classifier ensemble , 2018, Soft Comput..

[54]  de M.M.A. Graaf,et al.  Long-term evaluation of a social robot in real homes , 2014, HRI 2014.

[55]  Jason M. O'Kane,et al.  Adaptive Robot Design with Hand and Face Tracking for Use in Autism Therapy , 2010, ICSR.

[56]  Ahmad Lotfi,et al.  Socially Assistive Robotics: Robot Exercise Trainer for Older Adults , 2018 .

[57]  Danijela Ristic-Durrant,et al.  Robust Stereo-Vision Based 3D Object Reconstruction for the Assistive Robot FRIEND , 2011 .

[58]  Xin Pan,et al.  An Intention-Driven Semi-autonomous Intelligent Robotic System for Drinking , 2017, Front. Neurorobot..

[59]  Martin Jägersand,et al.  VIBI: Assistive vision-based interface for robot manipulation , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[60]  Geng Yang,et al.  cGAN Based Facial Expression Recognition for Human-Robot Interaction , 2019, IEEE Access.

[61]  Anselmo Frizera-Neto,et al.  Robot-Assisted Diagnosis for Children with Autism Spectrum Disorder Based on Automated Analysis of Nonverbal Cues , 2018, 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob).

[62]  Kimitoshi Yamazaki,et al.  A vision system for daily assistive robots using character information in daily environments , 2013, Proceedings of the 2013 IEEE/SICE International Symposium on System Integration.

[63]  Angel P. del Pobil,et al.  Object Detection and Recognition for Assistive Robots: Experimentation and Implementation , 2017, IEEE Robotics & Automation Magazine.

[64]  Eugenio Orosco,et al.  Robust Human Machine Interface Based on Head Movements Applied to Assistive Robotics , 2013, TheScientificWorldJournal.

[65]  Sergio Escalera,et al.  A real-time Human-Robot Interaction system based on gestures for assistive scenarios , 2016, Comput. Vis. Image Underst..

[66]  Claudia Di Napoli,et al.  A Framework for Personalized and Adaptive Socially Assistive Robotics , 2018, WOA.

[67]  Zhi Yan,et al.  Thermal Camera Based Physiological Monitoring with an Assistive Robot , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[68]  Mark Elshaw,et al.  A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots , 2018, Neural Computing and Applications.

[69]  Sao Mai Nguyen,et al.  Computational architecture of a robot coach for physical exercises in kinaesthetic rehabilitation , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[70]  Balasubramanian Thiagarajan,et al.  Modeling engagement in long-term, in-home socially assistive robot interventions for children with autism spectrum disorders , 2020, Science Robotics.

[71]  Jason M. O'Kane,et al.  CHARLIE : An Adaptive Robot Design with Hand and Face Tracking for Use in Autism Therapy , 2011, Int. J. Soc. Robotics.

[72]  Himalaya Patel,et al.  Toward Monitoring and Increasing Exercise Adherence in Older Adults by Robotic Intervention: A Proof of Concept Study , 2011, J. Robotics.

[73]  Yoshio Matsumoto,et al.  Assistive robotic arm autonomously bringing a cup to the mouth by face recognition , 2010, 2010 IEEE Workshop on Advanced Robotics and its Social Impacts.

[74]  Angel P. del Pobil,et al.  Vision for Robust Robot Manipulation , 2019, Sensors.

[75]  Jie Zhang,et al.  3D visual SLAM for an assistive robot in indoor environments using RGB-D cameras , 2014, 2014 9th International Conference on Computer Science & Education.

[76]  Masayuki Inaba,et al.  Transformable semantic map based navigation using autonomous deep learning object segmentation , 2016, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids).

[77]  Ioannis Brilakis,et al.  Real-time simulation of construction workers using combined human body and hand tracking for robotic construction worker system , 2018 .

[78]  Huiru Zheng,et al.  E-Bot: A Facial Recognition Based Human-Robot Emotion Detection System , 2018 .

[79]  Sibylle Meyer,et al.  Robotic companions in stroke therapy: A user study on the efficacy of assistive robotics among 30 patients in neurological rehabilitation , 2017, 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[80]  Maxime Raison,et al.  Fast scene analysis using vision and artificial intelligence for object prehension by an assistive robot , 2017, Eng. Appl. Artif. Intell..

[81]  Jindong Liu,et al.  Vision-based Automatic Control of a 5-Fingered Assistive Robotic Manipulator for Activities of Daily Living , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[82]  Vicente Mut,et al.  Vision-Based Interfaces Applied to Assistive Robots , 2013 .

[83]  Kamal M Othman,et al.  A Doorway Detection and Direction (3Ds) System for Social Robots via a Monocular Camera , 2020, Sensors.

[84]  Brenna Argall,et al.  Grasp detection for assistive robotic manipulation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[85]  Nicolás García Aracil,et al.  Multimodal Computer Vision Framework for Human Assistive Robotics , 2018, 2018 Workshop on Metrology for Industry 4.0 and IoT.

[86]  Yazarlar Gizlenmistir,et al.  Design and implementation of real time face tracking humanoid robot , 2018, 2018 26th Signal Processing and Communications Applications Conference (SIU).

[87]  Francisco Bellas,et al.  SARDAM: Service Assistant Robot for Daily Activity Monitoring , 2020, Proceedings.

[88]  Stavroula-Evita Fotinea,et al.  User Evaluation of the MOBOT rollator type robotic mobility assistive device , 2017, PETRA.

[89]  Maja J. Mataric,et al.  Robot exercise instructor: A socially assistive robot system to monitor and encourage physical exercise for the elderly , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[90]  João Sena Esteves,et al.  Mirroring and recognizing emotions through facial expressions for a RoboKind platform , 2017, 2017 IEEE 5th Portuguese Meeting on Bioengineering (ENBENG).

[91]  Pablo Bustos,et al.  A real time and robust facial expression recognition and imitation approach for affective human-robot interaction using Gabor filtering , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[92]  Teodiano Freire Bastos Filho,et al.  Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction , 2019, Sensors.

[93]  David Filliat,et al.  Generic Object Discrimination for Mobile Assistive Robots Using Projective Light Diffusion , 2018, 2018 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[94]  Thomas B. Sheridan,et al.  Human–Robot Interaction , 2016, Hum. Factors.

[95]  Vicente Julián,et al.  PHAROS—PHysical Assistant RObot System , 2018, Sensors.

[96]  Abdel Mehsen Ahmad,et al.  Kinect-Based Moving Human Tracking System with Obstacle Avoidance , 2017 .

[97]  Thomas B. Moeslund,et al.  Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks , 2019, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[98]  Guangmin Sun,et al.  Utilization of Color-depth Combination Features and Multi-level Refinement CNN for Upper-limb Posture Recognition , 2018, ICIGP.

[99]  Berardina De Carolis,et al.  The Humanoid Robot NAO as Trainer in a Memory Program for Elderly People with Mild Cognitive Impairment , 2020, Int. J. Soc. Robotics.

[100]  Ali Meghdari,et al.  Human–Robot Facial Expression Reciprocal Interaction Platform: Case Studies on Children with Autism , 2018, International Journal of Social Robotics.

[101]  Hatice Kose-Bagci,et al.  Sign Recognition System for an Assistive Robot Sign Tutor for Children , 2020, Int. J. Soc. Robotics.

[102]  Filippo Cavallo,et al.  Novel Technological Solutions for Assessment, Treatment, and Assistance in Mild Cognitive Impairment , 2019, Front. Neuroinform..

[103]  Yang Wang,et al.  An egocentric vision based assistive co-robot , 2013, 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR).

[104]  Kang-Hyun Jo,et al.  Real-time Face Tracking for Human-Robot Interaction , 2018, 2018 International Conference on Information and Communication Technology Robotics (ICT-ROBOT).

[105]  M. Cylkowska-Nowak,et al.  THE ROBOTS IN ELDERLY CARE , 2015 .

[106]  Iasonas Kokkinos,et al.  Human Joint Angle Estimation and Gesture Recognition for Assistive Robotic Vision , 2016, ECCV Workshops.

[107]  Danijela Ristic-Durrant,et al.  Robust stereo-vision based 3D modelling of real-world objects for assistive robotic applications , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.