Wearable Augmented Reality Application for Shoulder Rehabilitation

Augmented reality (AR) technology is gaining popularity and scholarly interest in the rehabilitation sector because of the possibility to generate controlled, user-specific environmental and perceptual stimuli which motivate the patient, while still preserving the possibility to interact with the real environment and other subjects, including the rehabilitation specialist. The paper presents the first wearable AR application for shoulder rehabilitation, based on Microsoft HoloLens, with real-time markerless tracking of the user’s hand. Potentialities and current limits of commercial head-mounted displays (HMDs) are described for the target medical field, and details of the proposed application are reported. A serious game was designed starting from the analysis of a traditional rehabilitation exercise, taking into account HoloLens specifications to maximize user comfort during the AR rehabilitation session. The AR application implemented consistently meets the recommended target frame rate for immersive applications with HoloLens device: 60 fps. Moreover, the ergonomics and the motivational value of the proposed application were positively evaluated by a group of five rehabilitation specialists and 20 healthy subjects. Even if a larger study, including real patients, is necessary for a clinical validation of the proposed application, the results obtained encourage further investigations and the integration of additional technical features for the proposed AR application.

[1]  E. Noé,et al.  Effect of a mixed reality-based intervention on arm, hand, and finger function on chronic stroke , 2016, Journal of NeuroEngineering and Rehabilitation.

[2]  Joaquim A. Jorge,et al.  Augmented Reality for Rehabilitation Using Multimodal Feedback , 2015, REHAB.

[3]  R. Satava,et al.  Virtual Reality Simulation for the Operating Room: Proficiency-Based Training as a Paradigm Shift in Surgical Skills Training , 2005, Annals of surgery.

[4]  Hong Hua.  Enabling Focus Cues in Head-Mounted Displays , 2017 .

[5]  Sara Condino,et al.  How to Build a Patient-Specific Hybrid Simulator for Orthopaedic Open Surgery: Benefits and Limits of Mixed-Reality Using the Microsoft HoloLens , 2018, Journal of healthcare engineering.

[6]  Hossein Mousavi Hondori,et al.  Choice of Human–Computer Interaction Mode in Stroke Rehabilitation , 2016, Neurorehabilitation and neural repair.

[7]  Yee Mon Aung,et al.  Augmented Reality based Illusion System with biofeedback , 2014, 2nd Middle East Conference on Biomedical Engineering.

[8]  Stefano Cagnoni,et al.  Design of a Wearable Sensing System for Human Motion Monitoring in Physical Rehabilitation , 2013, Sensors.

[9]  Christian Sandor,et al.  BrightView: Increasing Perceived Brightness of Optical See-Through Head-Mounted Displays Through Unnoticeable Incident Light Reduction , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[10]  R. Baños,et al.  Transforming Experience: The Potential of Augmented Reality and Virtual Reality for Enhancing Personal and Clinical Change , 2016, Front. Psychiatry.

[11]  S Condino,et al.  Natale et. al.'s response to Stecco's fascial nomenclature editorial. , 2014, Journal of bodywork and movement therapies.

[12]  K. Walker-Bone,et al.  Shoulder disorders and occupation. , 2015, Best practice & research. Clinical rheumatology.

[13]  Rosanna Maria Viglialoro,et al.  Proof of Concept: VR Rehabilitation Game for People with Shoulder Disorders , 2019, AVR.

[14]  Weidong Huang,et al.  Recent Trends of Mobile Collaborative Augmented Reality Systems , 2014 .

[15]  Esteban Correa-Agudelo,et al.  Computer Imagery and Neurological Rehabilitation: On the Use of Augmented Reality in Sensorimotor Training to Step Up Naturally Occurring Cortical Reorganization in Patients Following Stroke , 2016, MMVR.

[16]  Rosanna Maria Viglialoro,et al.  Review of the Augmented Reality Systems for Shoulder Rehabilitation , 2019, Inf..

[17]  Yee Mon Aung,et al.  Augmented reality-based RehaBio system for shoulder rehabilitation , 2014, Int. J. Mechatronics Autom..

[18]  H Martin Schepers,et al.  Validation of wearable visual feedback for retraining foot progression angle using inertial sensors and an augmented reality headset , 2018, Journal of NeuroEngineering and Rehabilitation.

[19]  Matteo Bianchi,et al.  Tactile Augmented Reality for Arteries Palpation in Open Surgery Training , 2016, MIAR.

[20]  Holger Regenbrecht,et al.  Augmented reality applications in rehabilitation to improve physical outcomes , 2012 .

[21]  -. YeeMonAung.  A Novel Upper Limb Rehabilitation System with Self-Driven Virtual Arm Illusion , 2015 .

[22]  Peter Kazanzides,et al.  Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display , 2017, International Journal of Computer Assisted Radiology and Surgery.

[23]  Roseli de Deus Lopes,et al.  An augmented reality system for upper-limb post-stroke motor rehabilitation: a feasibility study , 2016, Disability and rehabilitation. Assistive technology.

[24]  Sara Condino,et al.  Perceptual Limits of Optical See-Through Visors for Augmented Reality Guidance of Manual Tasks , 2020, IEEE Transactions on Biomedical Engineering.

[25]  Daniel J. Thomas,et al.  Augmented reality in surgery: The Computer-Aided Medicine revolution. , 2016, International journal of surgery.

[26]  Fatemeh Salehahmadi,et al.  Grand Adventure of Augmented Reality in Landscape of Surgery , 2019, World journal of plastic surgery.

[27]  Constantinos Loukas,et al.  An integrated approach to endoscopic instrument tracking for augmented reality applications in surgical simulation training , 2013, The international journal of medical robotics + computer assisted surgery : MRCAS.

[28]  Xun Luo,et al.  Integration of Augmented Reality and Assistive Devices for Post-Stroke Hand Opening Rehabilitation , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[29]  Nassir Navab,et al.  Upbeat: Augmented Reality-Guided Dancing for Prosthetic Rehabilitation of Upper Limb Amputees , 2019, Journal of healthcare engineering.

[30]  Veronica Teichrieb,et al.  Poster: Improving motor rehabilitation process through a natural interaction based system using Kinect sensor , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[31]  Daniel Medeiros,et al.  SleeveAR: Augmented Reality for Rehabilitation using Realtime Feedback , 2016, IUI.

[32]  Maria Teresa Restivo,et al.  A Gamified Approach for Hand Rehabilitation Device , 2018, Int. J. Online Eng..

[33]  A. Carr,et al.  Shoulder pain: diagnosis and management in primary care , 2005, BMJ : British Medical Journal.

[34]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[35]  Keith Dodd,et al.  The Development of Augmented Reality to Enhance Minimally Invasive Surgery. , 2017, Surgical technology international.

[36]  Fabrizio Cutolo,et al.  AR visualization of "synthetic Calot's triangle" for training in cholecystectomy , 2016, BioMed 2016.

[37]  Fabrizio Cutolo,et al.  Augmented Reality to Improve Surgical Simulation: Lessons Learned Towards the Design of a Hybrid Laparoscopic Simulator for Cholecystectomy , 2019, IEEE Transactions on Biomedical Engineering.

[38]  Yasushi Yamauchi.  Smart Dry Lab: An Augmented Reality (AR) Based Surgical Training Box , 2014, MMVR.

[39]  S. Adamovich,et al.  Virtual reality-augmented rehabilitation for patients following stroke. , 2002, Physical therapy.

[40]  Helene Langevin.  Langevin's response to Stecco's fascial nomenclature editorial. , 2014, Journal of bodywork and movement therapies.

[41]  Hye Sun Lee,et al.  Three-Dimensional Augmented Reality System for Balance and Mobility Rehabilitation in the Elderly: A Randomized Controlled Trial , 2019, Cyberpsychology Behav. Soc. Netw..

[42]  Marina-Anca Cidotã,et al.  Patient-Tailored Augmented Reality Games for Assessing Upper Extremity Motor Impairments in Parkinson’s Disease and Stroke , 2018, Journal of Medical Systems.

[43]  Leigh Hale,et al.  An affordable, computerised, table-based exercise system for stroke survivors , 2010, Disability and rehabilitation. Assistive technology.

[44]  Meng Ma,et al.  MirrARbilitation: A clinically-related gesture recognition interactive tool for an AR rehabilitation system , 2016, Comput. Methods Programs Biomed..

[45]  Dimitrios Chytas,et al.  Augmented Reality in Orthopedics: Current State and Future Directions , 2019, Front. Surg..

[46]  George Drettakis,et al.  Accommodation and Comfort in Head-Mounted Displays , 2018 .

[47]  Justin Dunn,et al.  Virtual and augmented reality in the treatment of phantom limb pain: A literature review. , 2017, NeuroRehabilitation.

[48]  Dmitry Oleynikov,et al.  Evaluation of Augmented Reality Feedback in Surgical Training Environment , 2018, Surgical innovation.

[49]  Sara Condino,et al.  Is the cervical fascia an anatomical proteus? , 2015, Surgical and Radiologic Anatomy.

[50]  Adel Al-Jumaily,et al.  Augmented reality game based multi-usage rehabilitation therapist for stroke patients , 2014 .