Real-time advanced spinal surgery via visible patient model and augmented reality system

This paper presents an advanced augmented reality system for spinal surgery assistance, and develops entry-point guidance prior to vertebroplasty spinal surgery. Based on image-based marker detection and tracking, the proposed camera-projector system superimposes pre-operative 3-D images onto patients. The patients' preoperative 3-D image model is registered by projecting it onto the patient such that the synthetic 3-D model merges with the real patient image, enabling the surgeon to see through the patients' anatomy. The proposed method is much simpler than heavy and computationally challenging navigation systems, and also reduces radiation exposure. The system is experimentally tested on a preoperative 3D model, dummy patient model and animal cadaver model. The feasibility and accuracy of the proposed system is verified on three patients undergoing spinal surgery in the operating theater. The results of these clinical trials are extremely promising, with surgeons reporting favorably on the reduced time of finding a suitable entry point and reduced radiation dose to patients.

[1]  M. Feuerstein.  Augmented Reality in Laparoscopic Surgery , 2007 .

[2]  N. Hurtós,et al.  Plane-based calibration of a projector-camera system , 2008 .

[3]  Daniela Gorski Trevisan,et al.  Towards an integrated system for planning and assisting maxillofacial orthognathic surgery , 2008, Comput. Methods Programs Biomed..

[4]  Jie Yang,et al.  Fast collision detection based on nose augmentation virtual surgery , 2007, Comput. Methods Programs Biomed..

[5]  D. Simon,et al.  Virtual Fluoroscopy: Computer-Assisted Fluoroscopic Navigation , 2001, Spine.

[6]  Bernhard Preim,et al.  Improved Navigated Spine Surgery Utilizing Augmented Reality Visualization , 2011, Bildverarbeitung für die Medizin.

[7]  Joonwhoan Lee,et al.  A Lighting Insensitive Face Detection Method on Color Images , 2012, 2012 Spring Congress on Engineering and Technology.

[8]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.

[9]  Neil D. Glossop,et al.  Laser projection augmented reality system for computer-assisted surgery , 2003, CARS.

[10]  HighWire Press,et al.  The journal of bone and joint surgery - British volume , 1948 .

[11]  Jean-Baptiste Fasquel,et al.  A modular and evolutive component oriented software architecture for patient modeling , 2006, Comput. Methods Programs Biomed..

[12]  Huei-Yung Lin,et al.  Geometric constraints for robot navigation using omnidirectional camera , 2012, 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[13]  J. Marescaux,et al.  Augmented Reality for Surgery and Interventional Therapy , 2005 .

[14]  Luc Soler,et al.  An augmented reality system to guide radio-frequency tumour ablation: Research Articles , 2005 .

[15]  Christian Nøhr,et al.  Comparing Approaches to Measuring the Adoption and Usability of Electronic Health Records: Lessons Learned from Canada, Denmark and Finland , 2013, MedInfo.

[16]  Nassir Navab,et al.  Action- and Workflow-Driven Augmented Reality for Computer-Aided Medical Procedures , 2007, IEEE Computer Graphics and Applications.

[17]  P. Kelly,et al.  Computer-assisted stereotaxic laser resection of intra-axial brain neoplasms. , 1986, Journal of neurosurgery.

[18]  Jung-Hwan Oh,et al.  Automatic real-time detection of endoscopic procedures using temporal features , 2012, Comput. Methods Programs Biomed..

[19]  Qionghai Dai,et al.  Fusing Multiview and Photometric Stereo for 3D Reconstruction under Uncalibrated Illumination , 2011, IEEE Transactions on Visualization and Computer Graphics.

[20]  Thomas Hartkens,et al.  The Hybrid Operating Room , 2012 .

[21]  Marcos Louredo,et al.  Bone drilling methodology and tool based on position measurements , 2013, Comput. Methods Programs Biomed..

[22]  Gabriel Taubin,et al.  Simple, Accurate, and Robust Projector-Camera Calibration , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[23]  G. Nollert,et al.  Planning a cardiovascular hybrid operating room: the technical point of view. , 2009, The heart surgery forum.

[24]  Heinz Wörn,et al.  Visualization of surgical 3D information with projector-based augmented reality. , 2005, Studies in health technology and informatics.

[25]  W. Eric L. Grimson,et al.  Evaluating and Validating an Automated Registration System for Enhanced Reality Visualization in Surgery , 1995, CVRMed.

[26]  Luc Soler,et al.  An augmented reality system to guide radio‐frequency tumour ablation , 2005, Comput. Animat. Virtual Worlds.

[27]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Ieee Robotics,et al.  IEEE journal of robotics and automation , 1985 .

[29]  Brian P. Bailey,et al.  Build your world and play in it: Interacting with surface particles on complex objects , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[30]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[31]  Nassir Navab,et al.  Camera-Augmented Mobile C-arm (CAMC) Application: 3D Reconstruction Using a Low-Cost Mobile C-arm , 1999, MICCAI.

[32]  Ekkehard Euler,et al.  Augmented reality visualization for thoracoscopic spine surgery , 2006, SPIE Medical Imaging.

[33]  Juan Carlos Fraile Marinero,et al.  A minimally invasive surgery robotic assistant for HALS-SILS techniques , 2013, Comput. Methods Programs Biomed..

[34]  Nassir Navab,et al.  Visualization and GPU-accelerated simulation of medical ultrasound from CT images , 2009, Comput. Methods Programs Biomed..

[35]  Marc Levoy,et al.  Reconstructing Occluded Surfaces Using Synthetic Apertures: Stereo, Focus and Robust Measures , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[36]  Pat Hanrahan,et al.  Volume Rendering , 2020, Definitions.

[37]  Soon-Yong Park,et al.  Active Calibration of Camera-Projector Systems Based on Planar Homography , 2010, 2010 20th International Conference on Pattern Recognition.

[38]  Tianmiao Wang,et al.  3D navigation and monitoring for spinal milling operation based on registration between multiplanar fluoroscopy and CT images , 2012, Comput. Methods Programs Biomed..

[39]  J. Dansereau,et al.  A points-to-surfaces matching technique for the application of augmented reality during spine surgery , 1995, Proceedings of 17th International Conference of the Engineering in Medicine and Biology Society.

[40]  Nord Catharina.  The visible patient , 2003 .

[41]  A.J.N. van Breemen,et al.  Design and implementation of a room thermostat using an agent-based approach , 2001 .

[42]  D. Hill,et al.  Augmentation of reality using an operating microscope for otolaryngology and neurosurgical guidance. , 1995, Journal of image guided surgery.

[43]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[44]  Klaus Radermacher,et al.  EVALUATION OF A 2D-FLUOROSCOPY-BASED NAVIGATION SYSTEM ENABLING A VIRTUAL RADIATION-FREE PREVIEW OF X-RAY IMAGES FOR INSERTION OF A PROXIMAL FEMORAL NAIL (PFNA®): AN EXPERIMENTAL STUDY , 2012 .

[45]  Ching-Chi Hsu,et al.  Comparison of multiple linear regression and artificial neural network in developing the objective functions of the orthopaedic screws , 2011, Comput. Methods Programs Biomed..