Real-Time Augmented Reality for Ear Surgery

Transtympanic procedures aim at accessing the middle ear structures through a puncture in the tympanic membrane. They require visualization of middle ear structures behind the eardrum. Up to now, this is provided by an oto endoscope. This work focused on implementing a real-time augmented reality based system for robotic-assisted transtympanic surgery. A preoperative computed tomography scan is combined with the surgical video of the tympanic membrane in order to visualize the ossciles and labyrinthine windows which are concealed behind the opaque tympanic membrane. The study was conducted on 5 artificial and 4 cadaveric temporal bones. Initially, a homography framework based on fiducials (6 stainless steel markers on the periphery of the tympanic membrane) was used to register a 3D reconstructed computed tomography image to the video images. Micro/endoscope movements were then tracked using Speeded-Up Robust Features. Simultaneously, a micro-surgical instrument (needle) in the frame was identified and tracked using a Kalman filter. Its 3D pose was also computed using a 3-collinear-point framework. An average initial registration accuracy of 0.21 mm was achieved with a slow propagation error during the 2-minute tracking. Similarly, a mean surgical instrument tip 3D pose estimation error of 0.33 mm was observed. This system is a crucial first step towards keyhole surgical approach to middle and inner ears.

[1]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Jay B. West,et al.  Fiducial Point Placement and the Accuracy of Point-based, Rigid Body Registration , 2001, Neurosurgery.

[3]  Shan Liu,et al.  A Portable 3D Vision Coordinate Measurement System Using a Light Pen , 2005 .

[4]  Philippe C. Cattin,et al.  Markerless Endoscopic Registration and Referencing , 2006, MICCAI.

[5]  Philippe Zanne,et al.  Visual Servoing-Based Endoscopic Path Following for Robot-Assisted Laparoscopic Surgery , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[7]  H. Hildmann,et al.  Approaches to the Middle Ear , 2006 .

[8]  M. Figl,et al.  2D/3D registration of endoscopic ultrasound to CT volume data , 2008, Physics in medicine and biology.

[9]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[10]  James K. Hahn,et al.  Registration of 3D CT Data to 2D Endoscopic Image using a Gradient Mutual Information based Viewpoint Matching for Image-Guided Medialization Laryngoplasty , 2010, J. Comput. Sci. Eng..

[11]  Guo Xiao-jun,et al.  Feature points based image registration between endoscope image and the CT image , 2011, 2011 International Conference on Electric Information and Control Engineering.

[12]  Zhoobin A. Mood,et al.  Use of a Microendoscope for Transtympanic Drug Delivery to the Round Window Membrane in Chinchillas , 2012, Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology.

[13]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[14]  Sven Haase,et al.  Laparoscopic instrument localization using a 3-D Time-of-Flight/RGB endoscope , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[15]  Max A. Viergever,et al.  A survey of medical image registration - under review , 2016, Medical Image Anal..

[16]  Marc Dean,et al.  Eustachian Tube Dilation via a Transtympanic Approach in 6 Cadaver Heads , 2016, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[17]  Alain Lalande,et al.  Augmented Reality of the Middle Ear Combining Otoendoscopy and Temporal Bone Computed Tomography. , 2018, Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology.