How to Build a Patient-Specific Hybrid Simulator for Orthopaedic Open Surgery: Benefits and Limits of Mixed-Reality Using the Microsoft HoloLens

Orthopaedic simulators are popular in innovative surgical training programs, where trainees gain procedural experience in a safe and controlled environment. Recent studies suggest that an ideal simulator should combine haptic, visual, and audio technology to create an immersive training environment. This article explores the potentialities of mixed-reality using the HoloLens to develop a hybrid training system for orthopaedic open surgery. Hip arthroplasty, one of the most common orthopaedic procedures, was chosen as a benchmark to evaluate the proposed system. Patient-specific anatomical 3D models were extracted from a patient computed tomography to implement the virtual content and to fabricate the physical components of the simulator. Rapid prototyping was used to create synthetic bones. The Vuforia SDK was utilized to register virtual and physical contents. The Unity3D game engine was employed to develop the software allowing interactions with the virtual content using head movements, gestures, and voice commands. Quantitative tests were performed to estimate the accuracy of the system by evaluating the perceived position of augmented reality targets. Mean and maximum errors matched the requirements of the target application. Qualitative tests were carried out to evaluate workload and usability of the HoloLens for our orthopaedic simulator, considering visual and audio perception and interaction and ergonomics issues. The perceived overall workload was low, and the self-assessed performance was considered satisfactory. Visual and audio perception and gesture and voice interactions obtained a positive feedback. Postural discomfort and visual fatigue obtained a nonnegative evaluation for a simulation session of 40 minutes. These results encourage using mixed-reality to implement a hybrid simulator for orthopaedic open surgery. An optimal design of the simulation tasks and equipment setup is required to minimize the user discomfort. Future works will include Face Validity, Content Validity, and Construct Validity to complete the assessment of the hip arthroplasty simulator.

[1]  E. Friis,et al.  Fatigue Performance of Composite Analogue Femur Constructs under High Activity Loading , 2007, Annals of Biomedical Engineering.

[2]  Peter Kazanzides,et al.  Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display , 2017, International Journal of Computer Assisted Radiology and Surgery.

[3]  Denise Nicholson.  Advances in Human Factors in Cybersecurity , 2016 .

[4]  Rebecca A. Grier.  How High is High? A Meta-Analysis of NASA-TLX Global Workload Scores , 2015 .

[5]  Arianna Menciassi,et al.  Patient Specific Virtual and Physical Simulation Platform for Surgical Robot Movability Evaluation in Single-Access Robot-Assisted Minimally-Invasive Cardiothoracic Surgery , 2017, AVR.

[6]  Ji-Sang Yoo,et al.  Qualitative analysis of individual and composite content factors of stereoscopic 3D video causing visual discomfort , 2013, Displays.

[7]  Gregory Kramida,et al.  Resolving the Vergence-Accommodation Conflict in Head-Mounted Displays , 2016, IEEE Transactions on Visualization and Computer Graphics.

[8]  Paolo Cignoni,et al.  MeshLab: an Open-Source Mesh Processing Tool , 2008, Eurographics Italian Chapter Conference.

[9]  M. Lorimer,et al.  What Is the Learning Curve for the Anterior Approach for Total Hip Arthroplasty? , 2015, Clinical orthopaedics and related research.

[10]  D. Gaba.  The future vision of simulation in health care , 2004, Quality and Safety in Health Care.

[11]  V Ferrari,et al.  How to build patient‐specific synthetic abdominal anatomies. An innovative approach from physical toward hybrid surgical simulators , 2011, The international journal of medical robotics + computer assisted surgery : MRCAS.

[12]  Fabrizio Cutolo,et al.  Augmented reality visualization of deformable tubular structures for surgical simulation , 2016, The international journal of medical robotics + computer assisted surgery : MRCAS.

[13]  C. Rorabeck,et al.  The operation of the century: total hip replacement , 2007, The Lancet.

[14]  Fabrizio Cutolo,et al.  AR visualization of "synthetic Calot's triangle" for training in cholecystectomy , 2016, BioMed 2016.

[15]  David Kane,et al.  The rate of change of vergence–accommodation conflict affects visual discomfort , 2014, Vision Research.

[16]  J. Callaghan,et al.  Adverse outcomes in hip arthroplasty: long-term trends. , 2012, The Journal of bone and joint surgery. American volume.

[17]  Sara Condino,et al.  Total Hip Replacement Simulators with Virtual Planning and Physical Replica for Surgical Training and Reharsal , 2016, BioMed 2016.

[18]  Sara Condino,et al.  Anthropomorphic ultrasound elastography phantoms — Characterization of silicone materials to build breast elastography phantoms , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[19]  Sara Condino,et al.  Computer tomography prototyping and virtual procedure simulation in difficult cases of hip replacement surgery. , 2013, Surgical technology international.

[20]  S. Botden,et al.  What is going on in augmented reality simulation in laparoscopic surgery? , 2008, Surgical Endoscopy.

[21]  Y. Hasegawa,et al.  SURGICAL SKILLS TRAINING FOR PRIMARY TOTAL HIP ARTHROPLASTY , 2015, Nagoya journal of medical science.

[22]  Woodrow Barfield,et al.  Fundamentals of Wearable Computers and Augumented Reality , 2000 .

[23]  Philip Tack,et al.  3D-printing techniques in a medical setting: a systematic literature review , 2016, BioMedical Engineering OnLine.

[24]  Jessica L Sparks,et al.  Use of Silicone Materials to Simulate Tissue Biomechanics as Related to Deep Tissue Injury , 2015, Advances in skin & wound care.

[25]  Allison M. Okamura,et al.  Modeling of Tool-Tissue Interactions for Computer-Based Surgical Simulation: A Literature Review , 2008, PRESENCE: Teleoperators and Virtual Environments.

[26]  Jens Grubert,et al.  A Survey of Calibration Methods for Optical See-Through Head-Mounted Displays , 2017, IEEE Transactions on Visualization and Computer Graphics.

[27]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[29]  E G Shifrin,et al.  Laparoscopic assisted aortic surgery. A review. , 2006, The Journal of cardiovascular surgery.

[30]  Sara Condino,et al.  Surgical simulators integrating virtual and physical anatomies , 2011, EICS4Med.

[31]  David M. Hoffman,et al.  Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. , 2008, Journal of vision.

[32]  Fabrizio Cutolo,et al.  Proof of Concept: Wearable Augmented Reality Video See-Through Display for Neuro-Endoscopy , 2018, AVR.

[33]  A. Heiner.  Structural properties of fourth-generation composite femurs and tibias. , 2008, Journal of biomechanics.

[34]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[35]  Neil Vaughan,et al.  A review of virtual reality based training simulators for orthopaedic surgery. , 2016, Medical engineering & physics.

[36]  P. Wooley,et al.  Mechanical Evaluation of Large-Size Fourth-Generation Composite Femur and Tibia Models , 2010, Annals of Biomedical Engineering.

[37]  Fabrizio Cutolo,et al.  [POSTER] Hybrid Video/Optical See-Through HMD , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[38]  Henry Fuchs,et al.  Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization , 2000, Presence: Teleoperators & Virtual Environments.

[39]  Rosanna Maria Viglialoro,et al.  Augmented Reality Simulator for Laparoscopic Cholecystectomy Training , 2014, AVR.

[40]  K. Ahmed,et al.  Current Status of Simulation-based Training Tools in Orthopedic Surgery: A Systematic Review. , 2017, Journal of surgical education.

[41]  Petter Andreas Steen,et al.  Manikins With Human-Like Chest Properties—A New Tool for Chest Compression Research , 2008, IEEE Transactions on Biomedical Engineering.