Intra-Operative Visualizations: Perceptual Fidelity and Human Factors

With increasing capability and complexity of surgical interventions, intra-operative visualization is becoming an important part of a surgical environment. This paper reviews some of our recent progress in the intelligent use of pre- and intra-operative data for enhanced surgical navigation and motion compensated visualization. High fidelity augmented reality (AR) with enhanced 3D depth perception is proposed to provide effective surgical guidance. To cater for large scale tissue deformation, real-time depth recovery based on stereo disparity and eye gaze tracking is introduced. This allows the development of motion compensated visualization for improved visual perception and for facilitating motion adaptive AR displays. The discussion of the paper is focused on how to ensure perceptual fidelity of AR and the need for real-time tissue deformation recovery and modeling, as well as the importance of incorporating human perceptual factors in surgical displays.

[1]  Frank Sauer,et al.  Reality Augmentation for Medical Procedures: System Architecture, Single Camera Marker Tracking, and System Evaluation , 2006, International Journal of Computer Vision.

[2]  Guang-Zhong Yang,et al.  Assessment of Perceptual Quality for Gaze-Contingent Motion Stabilization in Robotic Assisted Minimally Invasive Surgery , 2007, MICCAI.

[3]  Michael Figl,et al.  3D Reconstruction of Internal Organ Surfaces for Minimal Invasive Surgery , 2007, MICCAI.

[4]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[5]  Guang-Zhong Yang,et al.  Dense 3D Depth Recovery for Soft Tissue Deformation During Robotically Assisted Laparoscopic Surgery , 2004, MICCAI.

[6]  Guang-Zhong Yang,et al.  Stabilization of Image Motion for Robotic Assisted Beating Heart Surgery , 2007, MICCAI.

[7]  Tobias Ortmaier,et al.  Reconstruction of Image Structure in Presence of Specular Reflections , 2001, DAGM-Symposium.

[8]  Guang-Zhong Yang,et al.  pq-space Based Non-Photorealistic Rendering for Augmented Reality , 2007, MICCAI.

[9]  Ralph J Damiano.  Robotics in cardiac surgery: the Emperor's new clothes. , 2007, The Journal of thoracic and cardiovascular surgery.

[10]  George P. Mylonas,et al.  Gaze-contingent control for minimally invasive robotic surgery , 2006, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[11]  Guang-Zhong Yang,et al.  Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures , 2005, MICCAI.

[12]  Guang-Zhong Yang,et al.  Laparoscope Self-calibration for Robotic Assisted Minimally Invasive Surgery , 2005, MICCAI.

[13]  Darius Burschka,et al.  Scale-Invariant Registration of Monocular Endoscopic Images to CT-Scans for Sinus Surgery , 2004, MICCAI.

[14]  Gregory D. Hager,et al.  Stereo-Based Endoscopic Tracking of Cardiac Surface Deformation , 2004, MICCAI.

[15]  Guang-Zhong Yang,et al.  Eye-Gaze Driven Surgical Workflow Segmentation , 2007, MICCAI.

[16]  Russell H. Taylor,et al.  Medical robotics in computer-integrated surgery , 2003, IEEE Trans. Robotics Autom..

[17]  Guang-Zhong Yang,et al.  Patient-specific bronchoscope simulation with pq-space-based 2D/3D registration , 2004, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[18]  T. Peters.  Image-guidance for surgical procedures , 2006, Physics in medicine and biology.

[19]  Takayuki Okatani,et al.  3D Shape Reconstruction from an Endoscope Image Based on Shading , 1997 .

[20]  Hervé Delingette,et al.  Computational Models for Image-Guided Robot-Assisted and Simulated Medical Interventions , 2006, Proceedings of the IEEE.

[21]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[22]  Guang-Zhong Yang,et al.  Visual search: psychophysical models, practical applications , 2002, Image Vis. Comput..

[23]  Guang-Zhong Yang,et al.  Patient-specific bronchoscopy visualization through BRDF estimation and disocclusion correction , 2006, IEEE Transactions on Medical Imaging.

[24]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Takeo Kanade,et al.  When Is the Shape of a Scene Unique Given Its Light-Field: A Fundamental Theorem of 3D Vision? , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Guang-Zhong Yang,et al.  The use of visual search for knowledge gathering in image decision support , 2002, IEEE Transactions on Medical Imaging.

[27]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  Nassir Navab,et al.  Depth Perception - A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons , 2006, MICCAI.

[29]  Tobias Ortmaier,et al.  Tracking local motion on the beating heart , 2002, SPIE Medical Imaging.

[30]  Jun Sugiyama,et al.  Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images , 2002, Medical Image Anal..

[31]  H. Togami,et al.  Affects on visual search performance of individual differences in fixation time and number of fixations , 1984 .

[32]  Guang-Zhong Yang,et al.  Hot spot detection based on feature space representation of visual search , 2003, IEEE Transactions on Medical Imaging.

[33]  Berthold K. P. Horn,et al.  Shape from shading , 1989 .

[34]  Guang-Zhong Yang,et al.  Removing specular reflection components for robotic assisted laparoscopic surgery , 2005, IEEE International Conference on Image Processing 2005.

[35]  Peter Burger,et al.  Differential algorithm for the determination of shape from shading using a point light source , 1992, Image Vis. Comput..

[36]  Danail Stoyanov,et al.  A practical approach towards accurate dense 3D depth recovery for robotic laparoscopic surgery , 2005, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[37]  Guang-Zhong Yang,et al.  A gaze-based study for investigating the perception of visual realism in simulated scenes , 2008, TAP.