Auto-calibration Method for Active 3D Endoscope System Using Silhouette of Pattern Projector

In this paper, we develop an active stereo system for endoscope which requires auto-calibration, because a micro pattern projector is inserted through the instrument channel during an operation and cannot be fixed to the endoscope. For solution, a new auto-calibration technique with full 6-DOF estimation of an active stereo system without any extra devices nor extra pattern projections is proposed. In the technique, the pattern projector itself is simultaneously captured with a target scene by an endoscope camera and the silhouette of the pattern projector is used to conduct 2D-3D matching by using the knowledge of the shape of the projector. In addition, the markers which is included in the projection pattern are extracted and the distances from the closest epipolar lines are calculated as for the cost function. To enhance the robustness of the reconstruction, we also propose a simple high dynamic range (HDR) imaging system for an endoscope by alternating the input power of the pattern projector ON and OFF to blink the pattern so that exposure time will vary with beat frequency, realizing a virtual multi-exposure camera. By applying our auto calibration technique with HDR imaging system, we achieved a robust and accurate reconstruction of tissue in metric 3D under practical operation of the endoscopic system, such as reconstruction of the inside of a real stomach of a pig.

[1]  Eli Shechtman,et al.  Patch-based high dynamic range video , 2013, ACM Trans. Graph..

[2]  Ryo Furukawa,et al.  Self-Calibration of Multiple Laser Planes for 3D Scene Reconstruction , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[3]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[4]  Masashi Baba,et al.  2-DOF auto-calibration for a 3D endoscope system based on active stereo , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[5]  Ryo Furukawa,et al.  Laser range scanner based on self-calibration techniques using coplanarities and metric constraints , 2009, Comput. Vis. Image Underst..

[6]  Rafal Mantiuk,et al.  A comparative review of tone‐mapping algorithms for high dynamic range video , 2017, Comput. Graph. Forum.

[7]  Guang-Zhong Yang,et al.  Metric depth recovery from monocular images using Shape-from-Shading and specularities , 2012, 2012 19th IEEE International Conference on Image Processing.

[8]  Lena Maier-Hein,et al.  ToF Meets RGB: Novel Multi-Sensor Super-Resolution for Hybrid 3-D Endoscopy , 2013, MICCAI.

[9]  Shinji Tanaka,et al.  3D endoscope system using DOE projector , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[10]  Daniel S. Elson,et al.  An endoscopic structured light system using multispectral detection , 2015, International Journal of Computer Assisted Radiology and Surgery.

[11]  Ryo Furukawa,et al.  Uncalibrated multiple image stereo system with arbitrarily movable camera and projector for wide range scanning , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[12]  Christoph Schmalz,et al.  An endoscopic 3D scanner based on structured light , 2012, Medical Image Anal..

[13]  Yasushi Yagi,et al.  Dense 3D reconstruction method using a single pattern for fast moving object , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[14]  Guang-Zhong Yang,et al.  Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery , 2010, MICCAI.

[15]  Takeo Kanade,et al.  Simultaneous self-calibration of a projector and a camera using structured light , 2011, CVPR 2011 WORKSHOPS.

[16]  Shinji Tanaka,et al.  Calibration of a 3D endoscopic system based on active stereo method for shape measurement of biological tissues and specimen , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[17]  T. Nagakura,et al.  The study of three-dimensional measurement from an endoscopic images with stereo matching method , 2006, 2006 World Automation Congress.