Visual-Haptic Colocation in Robotic Rehabilitation Exercises Using a 2D Augmented-Reality Display

Haptics-based Virtual Reality (VR) games have been found to be effective in rehabilitation from disability. Augmented Reality (AR) has gained traction in recent years in various domains including gaming, entertainment, and education. In this paper, we integrate spatial AR into robotic rehabilitation to provide colocation between visual and haptic feedback as a human user participates in a rehabilitative game. A comparison between the effectiveness of VR vs AR (i.e., non-colocation vs colocation of vision) is done. Spatial AR is the colocation of vision through the use of projection. Visual-Haptic colocation is the combination of spatial AR and haptic interaction. We also compare each visualization technique in the absence and presence of haptic feedback and cognitive loading (CL) for the human user. The system was evaluated by having 10 able-bodied participants do all 8 different conditions lasting approximately 3 minutes per condition. The results show that spatial AR (corresponding to colocation of visual frame and hand frame) leads to the best user performance when doing the task regardless of the presence or the absence of haptics. It is also observed that for users undergoing cognitive loading, the combination of spatial AR and haptics produces the best result in terms of task completion time.

[1]  Mahdi Tavakoli,et al.  Learning and robotic imitation of therapist's motion and force for post-disability rehabilitation , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[2]  Maria del Carmen Juan Lizandra,et al.  An Augmented Reality System for the Treatment of Phobia to Small Animals Viewed Via an Optical See-Through HMD: Comparison With a Similar System Viewed Via a Video See-Through HMD , 2011, Int. J. Hum. Comput. Interact..

[3]  R. Macko,et al.  Testing a Model of Post‐Stroke Exercise Behavior , 2006, Rehabilitation nursing : the official journal of the Association of Rehabilitation Nurses.

[4]  Mahdi Tavakoli,et al.  Robotic assistance for children with cerebral palsy based on learning from tele-cooperative demonstration , 2017, International Journal of Intelligent Robotics and Applications.

[5]  Cristina V. Lopes,et al.  A Spatial Augmented Reality Rehab System for Post-Stroke Hand Rehabilitation , 2013, MMVR.

[6]  Mariela Rance,et al.  An augmented reality home-training system based on the mirror training and imagery approach , 2014, Behavior research methods.

[7]  Etienne Burdet,et al.  SITAR: a system for independent task-oriented assessment and rehabilitation , 2017, Journal of rehabilitation and assistive technologies engineering.

[8]  Marco Mendoza,et al.  Development of a haptic interface for motor rehabilitation therapy using augmented reality , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[9]  Michael Recce,et al.  A virtual reality based exercise system for hand rehabilitation post-stroke: transfer to function , 2004, EMBC 2004.

[10]  Mahdi Tavakoli,et al.  A Bicycle Cranking Model for Assist-as-Needed Robotic Rehabilitation Therapy Using Learning From Demonstration , 2016, IEEE Robotics and Automation Letters.

[11]  W. Sturm,et al.  Neuropsychological assessment , 2007, Journal of Neurology.

[12]  Sri Hastuti Kurniawan,et al.  An immersive physical therapy game for stroke survivors , 2014, ASSETS.

[13]  Dorothy Strickland,et al.  The challenge of using virtual reality in telerehabilitation. , 2004, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[14]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[15]  Michael Katchabaw,et al.  Immersive Augmented Reality for Parkinson Disease Rehabilitation , 2014 .

[16]  Cristina V. Lopes,et al.  Comparing “pick and place” task in spatial Augmented Reality versus non-immersive Virtual Reality for rehabilitation setting , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[17]  W J van der Eerden,et al.  CAREN--Computer Assisted Rehabilitation Environment. , 1999, Studies in health technology and informatics.

[18]  David J. Reinkensmeyer,et al.  "If I can't do it once, why do it a hundred times?": Connecting volition to movement success in a virtual environment motivates people to exercise the arm after stroke , 2007, 2007 Virtual Rehabilitation.

[19]  Xun Luo,et al.  Integration of Augmented Reality and Assistive Devices for Post-Stroke Hand Opening Rehabilitation , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[20]  Antonio Frisoli,et al.  A force-feedback exoskeleton for upper-limb rehabilitation in virtual reality , 2009 .

[21]  Jeffrey M. Hausdorff,et al.  Effects of cognitive function on gait and dual tasking abilities in patients with Parkinson’s disease suffering from motor response fluctuations , 2010, Experimental Brain Research.

[22]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[23]  Hossein Mousavi Hondori,et al.  Haptic Augmented Reality to monitor human arm's stiffness in rehabilitation , 2012, 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences.

[24]  Darryl Charles,et al.  Augmented Reality Games for Upper-Limb Stroke Rehabilitation , 2009, 2010 Second International Conference on Games and Virtual Worlds for Serious Applications.

[25]  Ana Grasielle Dionísio Corrêa,et al.  GenVirtual: An Augmented Reality Musical Game for Cognitive and Motor Rehabilitation , 2007, 2007 Virtual Rehabilitation.

[26]  Hiroshi Enaida,et al.  Development and evaluation of a visual aid using see-through display for patients with retinitis pigmentosa , 2014, Japanese Journal of Ophthalmology.

[27]  Víctor H. Andaluz,et al.  Virtual Reality Integration with Force Feedback in Upper Limb Rehabilitation , 2016, ISVC.

[28]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[29]  Shih-Ching Yeh,et al.  A Study for the Application of Body Sensing in Assisted Rehabilitation Training , 2012, 2012 International Symposium on Computer, Consumer and Control.