Deformable three-dimensional model architecture for interactive augmented reality in minimally invasive surgery

BackgroundSurgical procedures have undergone considerable advancement during the last few decades. More recently, the availability of some imaging methods intraoperatively has added a new dimension to minimally invasive techniques. Augmented reality in surgery has been a topic of intense interest and research.MethodsAugmented reality involves usage of computer vision algorithms on video from endoscopic cameras or cameras mounted in the operating room to provide the surgeon additional information that he or she otherwise would have to recognize intuitively. One of the techniques combines a virtual preoperative model of the patient with the endoscope camera using natural or artificial landmarks to provide an augmented reality view in the operating room. The authors’ approach is to provide this with the least number of changes to the operating room. Software architecture is presented to provide interactive adjustment in the registration of a three-dimensional (3D) model and endoscope video.ResultsAugmented reality including adrenalectomy, ureteropelvic junction obstruction, and retrocaval ureter and pancreas was used to perform 12 surgeries. The general feedback from the surgeons has been very positive not only in terms of deciding the positions for inserting points but also in knowing the least change in anatomy.ConclusionsThe approach involves providing a deformable 3D model architecture and its application to the operating room. A 3D model with a deformable structure is needed to show the shape change of soft tissue during the surgery. The software architecture to provide interactive adjustment in registration of the 3D model and endoscope video with adjustability of every 3D model is presented.

[1]  Michael Figl,et al.  Image guidance for robotic minimally invasive coronary artery bypass , 2010, Comput. Medical Imaging Graph..

[2]  Tina Kapur,et al.  Challenges in image-guided therapy system design , 2007, NeuroImage.

[3]  N. Navab,et al.  Advanced Medical Displays: A Literature Review of Augmented Reality , 2008, Journal of Display Technology.

[4]  Michiel de Looze,et al.  Meeting diversity in ergonomics. , 2006, Applied ergonomics.

[5]  Axel Pinz,et al.  The integration of optical and magnetic tracking for multi-user augmented reality , 1999, Comput. Graph..

[6]  L. Chen,et al.  Using an image-guided navigation system for localization of small pulmonary nodules before thoracoscopic surgery , 2007, Surgical Endoscopy.

[7]  B. Lopresti,et al.  Implementation and performance of an optical motion tracking system for high resolution brain PET imaging , 1998, 1998 IEEE Nuclear Science Symposium Conference Record. 1998 IEEE Nuclear Science Symposium and Medical Imaging Conference (Cat. No.98CH36255).

[8]  James S. Duncan,et al.  Development of a research interface for image guided intervention: initial application to epilepsy neurosurgery , 2006, 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 2006..

[9]  Michael W. Vannier,et al.  The operating room and the need for an IT infrastructure and standards , 2006, International Journal of Computer Assisted Radiology and Surgery.

[10]  Robert Lee Galloway.  Medical imaging 2003 : visualization, image-guided procedures, and display : 16-18 February 2003, San Diego, California, USA , 2003 .

[11]  M. Feuerstein,et al.  Navigation in endoscopic soft tissue surgery: perspectives and limitations. , 2008, Journal of endourology.

[12]  Jean-Baptiste Fasquel,et al.  A design pattern coupling role and component concepts: Application to medical software , 2011, J. Syst. Softw..

[13]  Christine DeLorenzo,et al.  From medical image computing to computer‐aided intervention: development of a research interface for image‐guided navigation , 2009, The international journal of medical robotics + computer assisted surgery : MRCAS.

[14]  A. Deguet,et al.  The cisst libraries for computer assisted intervention systems , 2008, The MIDAS Journal.

[15]  Su-Lin Lee,et al.  From medical images to minimally invasive intervention: Computer assistance for robotic surgery , 2010, Comput. Medical Imaging Graph..

[16]  J. Marescaux,et al.  A role-based component architecture for computer assisted interventions: illustration for electromagnetic tracking and robotized motion rejection in flexible endoscopy , 2009, The MIDAS Journal.

[17]  Heinz-Otto Peitgen,et al.  Illustrative visualization of 3D planning models for augmented reality in liver surgery , 2010, International Journal of Computer Assisted Radiology and Surgery.

[18]  Marco Nolden,et al.  The Medical Imaging Interaction Toolkit , 2004, Medical Image Anal..

[19]  W. Brent Seales,et al.  Endoscopic Video Texture Mapping on Pre-Built 3-D Anatomical Objects Without Camera Tracking , 2010, IEEE Transactions on Medical Imaging.

[20]  Marco Nolden,et al.  The extensible open-source rigid and affine image registration module of the Medical Imaging Interaction Toolkit (MITK) , 2010, Comput. Methods Programs Biomed..

[21]  Nassir Navab,et al.  Predicting and estimating the accuracy of n-occular optical tracking systems , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[22]  Adrien Bartoli,et al.  Deformable Shape-From-Motion in Laparoscopy using a Rigid Sliding Window , 2011, MIUA.

[23]  Judith Kremser,et al.  Comparison of different 3D navigation systems by a clinical “user” , 2001, European Archives of Oto-Rhino-Laryngology.

[24]  Ziv Yaniv,et al.  The Image-Guided Surgery Toolkit IGSTK: An Open Source C++ Software Toolkit , 2007, Journal of Digital Imaging.

[25]  Megumi Nakao,et al.  Physics-Based Interactive Volume Manipulation for Sharing Surgical Process , 2010, IEEE Transactions on Information Technology in Biomedicine.

[26]  R. Higuchi,et al.  Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery , 2010, Journal of hepato-biliary-pancreatic sciences.

[27]  Ron Kikinis,et al.  3D Slicer , 2012, 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro (IEEE Cat No. 04EX821).

[28]  Alan Liu,et al.  A Survey of Surgical Simulation: Applications, Technology, and Education , 2003, Presence: Teleoperators & Virtual Environments.

[29]  Guang-Zhong Yang,et al.  Soft tissue tracking for minimally invasive surgery: learning local deformation online. , 2008, Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention.

[30]  G. F. Buess,et al.  Comparative study of two-dimensional and three-dimensional vision systems for minimally invasive surgery , 1998, Surgical Endoscopy.

[31]  Alexandre Hostettler,et al.  A Cost Effective Simulator for Education of Ultrasound Image Interpretation and Probe Manipulation , 2011, MMVR.

[32]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[33]  Iker Aguinaga,et al.  Filtering of high modal frequencies for stable real-time explicit integration of deformable objects using the Finite Element Method. , 2010, Progress in biophysics and molecular biology.

[34]  Ashis Jalote-Parmar,et al.  Workflow Integration Matrix: a framework to support the development of surgical information systems , 2008 .

[35]  A. Bozorg Grayeli,et al.  Assessing mental representation of mastoidectomy by a computer-based drawing tool , 2010, Acta oto-laryngologica.

[36]  Ron Kikinis,et al.  Workflow modeling and analysis of computer guided prostate brachytherapy under MR imaging control. , 2004, Studies in health technology and informatics.

[37]  O. Ratib,et al.  Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion , 2011, Journal of hepato-biliary-pancreatic sciences.

[38]  Richard E. Clark,et al.  Intra-operative decision making: More than meets the eye , 2011, J. Biomed. Informatics.

[39]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.