Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery

This paper presents a method for real-time augmented reality of internal liver structures during minimally invasive hepatic surgery. Vessels and tumors computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to current methods, our method is able to locate the in-depth positions of the tumors based on partial three-dimensional liver tissue motion using a real-time biomechanical model. This model permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Experimentations conducted on phantom liver permits to measure the accuracy of the augmentation while real-time augmentation on in vivo human liver during real surgery shows the benefits of such an approach for minimally invasive surgery.

[1]  Gian Luca Mariottini,et al.  Toward long-term and accurate Augmented-Reality display for minimally-invasive surgery , 2013, 2013 IEEE International Conference on Robotics and Automation.

[2]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Lena Maier-Hein,et al.  Physics-based shape matching for intraoperative image guidance. , 2014, Medical physics.

[4]  Adrien Bartoli,et al.  Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  M. Freeman.  Strength of Biological Materials , 1971 .

[6]  Stephane Cotin,et al.  New Approaches to Catheter Navigation for Interventional Radiology Simulation , 2005, MICCAI.

[7]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[8]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[9]  Hideo Saito,et al.  Texture Overlay onto Non-rigid Surface using Commodity Depth Camera , 2012, VISAPP.

[10]  B. Edwin,et al.  A comparative study of the short-term outcome following open and laparoscopic liver resection of colorectal metastases , 2002, Surgical Endoscopy.

[11]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[12]  Amy E. Kerdok,et al.  Effects of perfusion on the viscoelastic characteristics of liver. , 2006, Journal of biomechanics.

[13]  Stephane Cotin,et al.  Modeling and Real-Time Simulation of a Vascularized Liver Tissue , 2012, MICCAI.

[14]  Patrick Clarysse,et al.  A dynamic elastic model for segmentation and tracking of the heart in MR image sequences , 2010, Medical Image Anal..

[15]  Nazim Haouchine,et al.  Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[16]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[17]  Carlos A. Felippa,et al.  A unified formulation of small-strain corotational finite elements: I. Theory , 2005 .

[18]  Guang-Zhong Yang,et al.  Dynamic Guidance for Robotic Surgery Using Image-Constrained Biomechanical Models , 2010, MICCAI.

[19]  Nazim Haouchine,et al.  Towards an accurate tracking of liver tumors for augmented reality in robotic assisted surgery , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[20]  Yuncai Liu,et al.  Monocular 3-D Tracking of Inextensible Deformable Surfaces Under $L_2$ -Norm , 2010, IEEE Transactions on Image Processing.

[21]  Stephane Cotin,et al.  Truth cube: Establishing physical standards for soft tissue simulation , 2003, Medical Image Anal..

[22]  Christian Duriez,et al.  SOFA: A Multi-Model Framework for Interactive Physical Simulation , 2012 .

[23]  Michael Figl,et al.  Image Guidance for Robotic Minimally Invasive Coronary Artery Bypass , 2008, MIAR.

[24]  Adrien Bartoli,et al.  Direct Estimation of Non-Rigid Registration , 2004, BMVC.

[25]  J. M. M. Montiel,et al.  Finite Element based sequential Bayesian Non-Rigid Structure from Motion , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Jochen Lang,et al.  Tracking Complete Deformable Objects with Finite Elements , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[27]  Yohan Payan,et al.  Efficient, Physically Plausible Finite Elements , 2005, Eurographics.

[28]  Adrien Bartoli,et al.  Tracking by Detection for Interactive Image Augmentation in Laparoscopy , 2012, WBIR.

[29]  Luc Soler,et al.  Experimental in vitro mechanical characterization of porcine Glisson's capsule and hepatic veins. , 2011, Journal of biomechanics.

[30]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.

[31]  J. L. Moyano-Cuevas,et al.  Anatomical changes due to pneumoperitoneum analyzed by MRI: an experimental study in pigs , 2011, Surgical and Radiologic Anatomy.

[32]  Lena Maier-Hein,et al.  Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery , 2013, Medical Image Anal..

[33]  Michael R. Lyu,et al.  Progressive Finite Newton Approach To Real-time Nonrigid Surface Detection , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Anna Hilsmann,et al.  Realistic cloth augmentation in single view video under occlusions , 2010, Comput. Graph..

[35]  Lena Maier-Hein,et al.  Comparative Validation of Single-Shot Optical Techniques for Laparoscopic 3-D Surface Reconstruction , 2014, IEEE Transactions on Medical Imaging.

[36]  Pascal Fua,et al.  Surface Deformation Models for Nonrigid 3D Shape Recovery , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Nazim Haouchine,et al.  Deformation-based Augmented Reality for Hepatic Surgery , 2013, MMVR.

[38]  Theodore Kim,et al.  Semi-automated soft-tissue acquisition and modeling for surgical simulation , 2009, 2009 IEEE International Conference on Automation Science and Engineering.

[39]  Stéphane Cotin,et al.  Truth Cube: Establishing Physical Standards for Real Time Soft Tissue Simulation , 2001 .

[40]  C. Duriez,et al.  New approaches to catheter navigation for interventional radiology simulation , 2006, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[41]  Patrick Bouthemy,et al.  Computation and analysis of image motion: A synopsis of current problems and methods , 1996, International Journal of Computer Vision.

[42]  Aleksandra Popovic,et al.  Robust feature tracking on the beating heart for a robotic‐guided endoscope , 2011, The international journal of medical robotics + computer assisted surgery : MRCAS.

[43]  Rémy Willinger,et al.  Multiplicative Jacobian Energy Decomposition Method for Fast Porous Visco-Hyperelastic Soft Tissue Model , 2010, MICCAI.

[44]  Hervé Delingette,et al.  Deformable biomechanical models: Application to 4D cardiac image analysis , 2003, Medical Image Anal..

[45]  J. S. Przemieniecki.  Theory of matrix structural analysis , 1985 .

[46]  Nazim Haouchine,et al.  Workshop on Virtual Reality Interaction and Physical Simulation Vriphys (2012) Physics-based Augmented Reality for 3d Deformable Object Author Manuscript, Published in "vriphys -virtual Reality Interaction and Physical Simulation (2012)" Physics-based Augmented Reality for 3d Deformable Object , 2022 .

[47]  Hongsheng Li,et al.  Active Volume Models for Medical Image Segmentation , 2011, IEEE Transactions on Medical Imaging.

[48]  Vincent Lepetit,et al.  Fast Non-Rigid Surface Detection, Registration and Realistic Augmentation , 2008, International Journal of Computer Vision.