Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation

Autostereoscopic 3D image overlay for augmented reality (AR) based surgical navigation has been studied and reported many times. For the purpose of surgical overlay, the 3D image is expected to have the same geometric shape as the original organ, and can be transformed to a specified location for image overlay. However, how to generate a 3D image with high geometric fidelity and quantitative evaluation of 3D image's geometric accuracy have not been addressed. This paper proposes a graphics processing unit (GPU) based computer-generated integral imaging pipeline for real-time autostereoscopic 3D display, and an automatic closed-loop 3D image calibration paradigm for displaying undistorted 3D images. Based on the proposed methods, a novel AR device for 3D image surgical overlay is presented, which mainly consists of a 3D display, an AR window, a stereo camera for 3D measurement, and a workstation for information processing. The evaluation on the 3D image rendering performance with 2560×1600 elemental image resolution shows the rendering speeds of 50-60 frames per second (fps) for surface models, and 5-8 fps for large medical volumes. The evaluation of the undistorted 3D image after the calibration yields sub-millimeter geometric accuracy. A phantom experiment simulating oral and maxillofacial surgery was also performed to evaluate the proposed AR overlay device in terms of the image registration accuracy, 3D image overlay accuracy, and the visual effects of the overlay. The experimental results show satisfactory image registration and image overlay accuracy, and confirm the system usability.

[1]  Mtm Marc Lambooij,et al.  Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review , 2009 .

[2]  Ken Masamune,et al.  Non-metal Slice Image Overlay Display System Used Inside the Open Type MRI , 2008, MIAR.

[3]  G. Lippmann.  Epreuves reversibles donnant la sensation du relief , 1908 .

[4]  J. Debus,et al.  Projector-based augmented reality for intuitive intraoperative guidance in image-guided 3D interstitial brachytherapy. , 2008, International journal of radiation oncology, biology, physics.

[5]  Sim Heng Ong,et al.  Projection-based visual guidance for robot-aided RF needle insertion , 2013, International Journal of Computer Assisted Radiology and Surgery.

[6]  Vincenzo Ferrari,et al.  A 3-D Mixed-Reality System for Stereoscopic Visualization of Medical Dataset , 2009, IEEE Transactions on Biomedical Engineering.

[7]  Ken Masamune,et al.  Image overlay guidance for needle insertion in CT scanner , 2005, IEEE Transactions on Biomedical Engineering.

[8]  Neil A. Dodgson,et al.  Autostereoscopic 3D displays , 2005, Computer.

[9]  A. Park,et al.  Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography , 2010, Surgical Endoscopy.

[10]  J Arai,et al.  Real-time pickup method for a three-dimensional image based on integral photography. , 1997, Applied optics.

[11]  Bahram Javidi,et al.  Three-Dimensional Image Sensing, Visualization, and Processing Using Integral Imaging , 2006, Proceedings of the IEEE.

[12]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.

[13]  B. Javidi,et al.  Integral three-dimensional imaging with digital reconstruction. , 2001, Optics letters.

[14]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[15]  Bahram Javidi,et al.  Progress in 3-D Multiperspective Display by Integral Imaging , 2009, Proceedings of the IEEE.

[16]  Julien Nauroy,et al.  Real time planning, guidance and validation of surgical acts using 3D segmentations, augmented reality projections and surgical tools video tracking , 2010, Medical Imaging.

[17]  Ken Masamune,et al.  Computerized Medical Imaging and Graphics , 2009 .

[18]  Hongen Liao,et al.  3-D Augmented Reality for MRI-Guided Surgery Using Integral Videography Autostereoscopic Image Overlay , 2010, IEEE Transactions on Biomedical Engineering.

[19]  Nobuhiko Hata,et al.  Three-dimensional display with a long viewing distance by use of integral photography. , 2005, Optics letters.

[20]  Nobuhiko Hata,et al.  Surgical navigation by autostereoscopic image overlay of integral videography , 2004, IEEE Transactions on Information Technology in Biomedicine.

[21]  Ichiro Sakuma,et al.  Augmented Reality Navigation With Automatic Marker-Free Image Registration Using 3-D Image Overlay for Dental Surgery , 2014, IEEE Transactions on Biomedical Engineering.

[22]  Byoungho Lee,et al.  Recent progress in three-dimensional information processing based on integral imaging. , 2009, Applied optics.

[23]  Wolfgang Birkfellner,et al.  A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus , 2005, IEEE Transactions on Medical Imaging.

[24]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[25]  Martin Kraus,et al.  High-quality pre-integrated volume rendering using hardware-accelerated pixel shading , 2001, HWWS '01.

[26]  Jay B. West,et al.  The distribution of target registration error in rigid-body point-based registration , 2001, IEEE Transactions on Medical Imaging.

[27]  Gilbae Park,et al.  Depth-enhanced integral imaging display system with electrically variable image planes using polymer-dispersed liquid-crystal layers. , 2007, Applied optics.

[28]  Bahram Javidi,et al.  Three-dimensional volumetric object reconstruction using computational integral imaging. , 2004, Optics express.