Research on Accuracy of Augmented Reality Surgical Navigation System Based on Multi-View Virtual and Real Registration Technology

The traditional navigation system used in spinal puncture is not able to monitor the surgical process in real time and the accuracy of navigation is unsatisfactory. In this study, an augmented reality surgical navigation system based on multi-view virtual and real registration is proposed to solve these problems. The theory of virtual and real registration in augmented reality technology is analyzed, and the methods of single-view and multi-view virtual and real registration are compared. The principle of coordinate transformation in the surgical navigation module is analyzed. The platform of augmented reality surgical navigation system based on multi-view virtual and real registration technology is designed. The experiments of the spinal model are used to verify the accuracy of virtual and real registration. The accuracy of the proposed navigation system is verified by the experiment of simulating puncture operation with robotic control of surgical tool. The experimental results show that the accuracy of single-view and multi-view virtual and real registration is 9.85±0.80mm and 1.62±0.22mm respectively. The accuracy of the augmented reality surgical navigation system added with multi-view virtual and real registration technology is 1.70±0.25mm, which is 35% higher than that of the augmented reality surgical navigation system not previously used. The augmented reality surgical navigation system for spinal puncture based on multi-view virtual and real registration proposed in this study can meet the requirements of physician on the accuracy of surgery. It can also help physician to monitor the surgical process in real time and improve the success rate of surgery.

[1]  Kai-Che Liu,et al.  Deformable three-dimensional model architecture for interactive augmented reality in minimally invasive surgery , 2012, Surgical Endoscopy.

[2]  Luc Soler,et al.  Robotic duodenopancreatectomy assisted with augmented reality and real-time fluorescence guidance , 2014, Surgical Endoscopy.

[3]  Wolfgang Friedrich,et al.  ARVIKA-augmented reality for development, production and service , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[4]  K. Weimer,et al.  Mixed Reality with HoloLens: Where Virtual Reality Meets Augmented Reality in the Operating Room , 2017, Plastic and reconstructive surgery.

[5]  Aleksandra Popovic,et al.  Feasibility and accuracy of a robotic guidance system for navigated spine surgery in a hybrid operating room: a cadaver study , 2020, Scientific Reports.

[6]  B. Jaramaz,et al.  Computer Assisted Orthopaedic Surgery: Image Guided and Robotic Assistive Technologies , 1998, Clinical orthopaedics and related research.

[7]  Graham Lawton,et al.  Through the HoloLens™ looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels , 2018, European Radiology Experimental.

[8]  Michael Söderman,et al.  Pedicle Screw Placement Using Augmented Reality Surgical Navigation With Intraoperative 3D Imaging , 2018, Spine.

[9]  Ken Masamune,et al.  An effective visualization technique for depth perception in augmented reality‐based surgical navigation , 2016, The international journal of medical robotics + computer assisted surgery : MRCAS.

[10]  Stephanie B. Jones,et al.  Augmented Reality and Ultrasound Education: Initial Experience. , 2017, Journal of cardiothoracic and vascular anesthesia.

[11]  Henry Fuchs,et al.  Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization , 2000, Presence: Teleoperators & Virtual Environments.

[12]  M. Farshad,et al.  Augmented reality-based navigation increases precision of pedicle screw insertion , 2020, Journal of Orthopaedic Surgery and Research.

[13]  Long Chen,et al.  Research on the accuracy of three‐dimensional localization and navigation in robot‐assisted spine surgery , 2019, The international journal of medical robotics + computer assisted surgery : MRCAS.

[14]  Shinsuk Park,et al.  Development of a surgical navigation system for corrective osteotomy based on augmented reality , 2017 .

[15]  Naoki Suzuki,et al.  Novel navigation system by augmented reality technology using a tablet PC for hepatobiliary and pancreatic surgery , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[16]  Manoranjan Paul,et al.  A novel augmented reality (AR) scheme for knee replacement surgery by considering cutting error accuracy , 2019, The international journal of medical robotics + computer assisted surgery : MRCAS.

[17]  Ken Masamune,et al.  Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study , 2013, International Journal of Oral Science.

[18]  Jang W Yoon,et al.  Augmented reality for the surgeon: Systematic review , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[19]  Nassir Navab,et al.  Action- and Workflow-Driven Augmented Reality for Computer-Aided Medical Procedures , 2007, IEEE Computer Graphics and Applications.

[20]  John Vince,et al.  Introduction to Virtual Reality , 2004, Springer London.

[21]  P. Dasgupta,et al.  Systematic review of augmented reality in urological interventions: the evidences of an impact on surgical outcomes are yet to come , 2019, World Journal of Urology.

[22]  Timothy N Judkins,et al.  Augmented reality and haptic interfaces for robot‐assisted surgery , 2012, The international journal of medical robotics + computer assisted surgery : MRCAS.

[23]  Gudrun Klinker,et al.  Semi-Parametric Color Reproduction Method for Optical See-Through Head-Mounted Displays , 2015, IEEE Transactions on Visualization and Computer Graphics.

[24]  G. A. Giraldi,et al.  Introduction to Augmented Reality , 2003 .

[25]  Manabu Ito,et al.  A novel 3D guidance system using augmented reality for percutaneous vertebroplasty: technical note. , 2013, Journal of neurosurgery. Spine.

[26]  N. Navab,et al.  Advanced Medical Displays: A Literature Review of Augmented Reality , 2008, Journal of Display Technology.

[27]  Horacio Rios,et al.  An introduction to Augmented Reality with applications in aeronautical maintenance , 2011, 2011 13th International Conference on Transparent Optical Networks.

[28]  K. Cleary,et al.  Image-guided interventions: technology review and clinical applications. , 2010, Annual review of biomedical engineering.

[29]  Shaodong Li,et al.  A Robot-Assisted Spine Surgery System Based on Intraoperative 2D Fluoroscopy Navigation , 2020, IEEE Access.

[30]  H. Liao,et al.  Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods. , 2018, Hepatobiliary & pancreatic diseases international : HBPD INT.

[31]  William Omar Contreras López,et al.  Intraoperative clinical application of augmented reality in neurosurgery: A systematic review , 2019, Clinical Neurology and Neurosurgery.

[32]  Hongen Liao,et al.  Three‐dimensional augmented reality surgical navigation with hybrid optical and electromagnetic tracking for distal intramedullary nail interlocking , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[33]  Vincenzo Denaro,et al.  Robotic Spine Surgery and Augmented Reality Systems: A State of the Art , 2020, Neurospine.

[34]  A. Vaccaro,et al.  Image-guidance, Robotics, and the Future of Spine Surgery , 2019, Clinical spine surgery.

[35]  Davide Scaramuzza,et al.  Pedicle screw navigation using surface digitization on the Microsoft HoloLens , 2019, International Journal of Computer Assisted Radiology and Surgery.

[36]  Christopher Nimsky,et al.  Microscope-Based Augmented Reality in Degenerative Spine Surgery: Initial Experience. , 2019, World neurosurgery.

[37]  Ramin Javan,et al.  Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography , 2018, International Journal of Computer Assisted Radiology and Surgery.

[38]  Ichiro Sakuma,et al.  Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation , 2015, Comput. Medical Imaging Graph..

[39]  Mohammed A Almekhlafi,et al.  Simulation and augmented reality in endovascular neurosurgery: lessons from aviation. , 2013, Neurosurgery.

[40]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[41]  Jan Egger,et al.  Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display , 2015, J. Biomed. Informatics.

[42]  M. Citardi,et al.  Augmented reality for endoscopic sinus surgery with surgical navigation: a cadaver study , 2016, International forum of allergy & rhinology.

[43]  Michael T. Pascale,et al.  Detection of visual stimuli on monocular peripheral head-worn displays. , 2018, Applied ergonomics.

[44]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[45]  Gudrun Klinker,et al.  Simultaneous Direct and Augmented View Distortion Calibration of Optical See-Through Head-Mounted Displays , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[46]  Tillman Weyde,et al.  3d augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support , 2007, ICMI '07.

[47]  Philip Pratt,et al.  Transoral Robotic Surgery: Image Guidance and Augmented Reality , 2018, ORL.

[48]  Nassir Navab,et al.  The Virtual Mirror: A New Interaction Paradigm for Augmented Reality Environments , 2009, IEEE Transactions on Medical Imaging.