Augmented Reality Visualization of Joint Movements for Rehabilitation and Sports Medicine

We present a visualization tool for human motion analysis in augmented reality. Our tool builds upon our previous work on joint biomechanical modelling for kinematic analysis, based on optical motion capture and personalized anatomical reconstruction of joint structures from medical imaging. It provides healthcare professionals with the in situ visualization of joint movements, where bones are accurately rendered as a holographic overlay on the subject – like if the user has an “X-ray vision” – and in real-time as the subject performs the movement. The tool also provides a recording mechanism for the examination and acquisition of movements and range of motion information. Recorded information can be for instance retrieved at a later moment to assess patient’s progress in terms of kinematics during the rehabilitation phase. We also propose an intuitive non-sequential mean of navigating through recordings. It consists of pointing at movement trajectories for easy and intuitive retrieval of the meaningful portions of a movement. This tool allows for the post hoc replay and analysis of fast movements, such as from athletes movements in sports injury evaluation. Currently, hip and knee joints are supported.

[1]  Nassir Navab,et al.  Single-Point Active Alignment Method (SPAAM) for Optical See-Through HMD Calibration for Augmented Reality , 2002, Presence: Teleoperators & Virtual Environments.

[2]  Tobias Höllerer,et al.  Evaluating gesture-based augmented reality annotation , 2017, 2017 IEEE Symposium on 3D User Interfaces (3DUI).

[3]  Greg Welch,et al.  SCAAT: incremental tracking with incomplete information , 1997, SIGGRAPH.

[4]  Long Chen,et al.  Recent Developments and Future Challenges in Medical Mixed Reality , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[5]  Jacky C. K. Chow,et al.  A patient-specific measurement technique to model shoulder joint kinematics. , 2014, Orthopaedics & traumatology, surgery & research : OTSR.

[6]  Hartmut Witte,et al.  ISB recommendation on definitions of joint coordinate system of various joints for the reporting of human joint motion--part I: ankle, hip, and spine. International Society of Biomechanics. , 2002, Journal of biomechanics.

[7]  Simon Weidert,et al.  Kinect for interactive AR anatomy learning , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[8]  Claus Gramkow,et al.  On Averaging Rotations , 2001, International Journal of Computer Vision.

[9]  Mark Billinghurst,et al.  A Survey of Augmented Reality , 2015, Found. Trends Hum. Comput. Interact..

[10]  Michael J. Griffin,et al.  Compensating Lags in Head-Coupled Displays Using Head Position Prediction and Image Deflection , 1992 .

[11]  Anderson Maciel,et al.  Spatially aware mobile interface for 3D visualization and interactive surgery planning , 2014, 2014 IEEE 3nd International Conference on Serious Games and Applications for Health (SeGAH).

[12]  Henry Been-Lirn Duh,et al.  Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[13]  Jun G San Juan,et al.  Concurrent validity of digital inclinometer and universal goniometer in assessing passive hip mobility in healthy subjects. , 2013, International journal of sports physical therapy.

[14]  M. Batt,et al.  A Review of Femoroacetabular Impingement in Athletes , 2008, Sports medicine.

[15]  Anderson Maciel,et al.  Anatomic Hepatectomy Planning through Mobile Display Visualization and Interaction , 2012, MMVR.

[16]  D. W. F. van Krevelen,et al.  A Survey of Augmented Reality Technologies, Applications and Limitations , 2010, Int. J. Virtual Real..

[17]  F. Wang,et al.  Precision insertion of percutaneous sacroiliac screws using a novel augmented reality-based navigation system: a pilot study , 2016, International Orthopaedics.

[18]  Nassir Navab,et al.  Depth Perception - A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons , 2006, MICCAI.

[19]  Meng Ma,et al.  MirrARbilitation: A clinically-related gesture recognition interactive tool for an AR rehabilitation system , 2016, Comput. Methods Programs Biomed..

[20]  A. Cappozzo,et al.  Human movement analysis using stereophotogrammetry. Part 3. Soft tissue artifact assessment and compensation. , 2005, Gait & posture.

[21]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Lena Maier-Hein,et al.  Mobile markerless augmented reality and its application in forensic medicine , 2015, International Journal of Computer Assisted Radiology and Surgery.

[23]  Stuart M McGill,et al.  Quantifying normal 3D hip ROM in healthy young adult males with clinical and laboratory tools: hip mobility restrictions appear to be plane-specific. , 2011, Clinical biomechanics.

[24]  Sander Oude Elberink,et al.  Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications , 2012, Sensors.

[25]  Sonia Duprey,et al.  Influence of joint constraints on lower limb kinematics estimation from skin markers using global optimization. , 2010, Journal of biomechanics.

[26]  Kuo-Chi Lin,et al.  Automatic Modeling of Knee-Joint Motion For The Virtual Reality Dynamic Anatomy (VRDA) Tool , 2000, Presence: Teleoperators & Virtual Environments.

[27]  Daniel J. Wigdor,et al.  AnatOnMe: facilitating doctor-patient communication using a projection-based handheld device , 2011, CHI.

[28]  Tomasz Mazuryk,et al.  Two‐step Prediction and Image Deflection for Exact Head Tracking in Virtual Environments , 1995, Comput. Graph. Forum.

[29]  Doug A. Bowman,et al.  An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments , 1997, SI3D.

[30]  Nassir Navab,et al.  Advanced training methods using an Augmented Reality ultrasound simulator , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[31]  R. Ganz,et al.  The Concept of Femoroacetabular Impingement: Current Status and Future Perspectives , 2009, Clinical orthopaedics and related research.

[32]  N. Navab,et al.  Advanced Medical Displays: A Literature Review of Augmented Reality , 2008, Journal of Display Technology.

[33]  E S Grood,et al.  A joint coordinate system for the clinical description of three-dimensional motions: application to the knee. , 1983, Journal of biomechanical engineering.

[34]  Nadia Magnenat-Thalmann,et al.  Assessment of Congruence and Impingement of the Hip Joint in Professional Ballet Dancers , 2011, The American journal of sports medicine.

[35]  Albert A. Rizzo,et al.  Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[36]  Caecilia Charbonnier,et al.  Does surgery for instability of the shoulder truly stabilize the glenohumeral joint? , 2016, Medicine.

[37]  Nassir Navab,et al.  mirracle: An augmented reality magic mirror system for anatomy education , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[38]  Manabu Ito,et al.  A novel 3D guidance system using augmented reality for percutaneous vertebroplasty: technical note. , 2013, Journal of neurosurgery. Spine.

[39]  Jacques Menetrey,et al.  Multi-body optimization with subject-specific knee models: performance at high knee flexion angles , 2017, Computer methods in biomechanics and biomedical engineering.

[40]  Nadia Magnenat-Thalmann,et al.  Motion study of the hip joint in extreme postures , 2009, The Visual Computer.

[41]  Peter Kazanzides,et al.  Comprehensive Tracker Based Display Calibration for Holographic Optical See-Through Head-Mounted Display , 2017, ArXiv.