Deep Endoscope: Intelligent Duct Inspection for the Avionic Industry

We present the first autonomous endoscope for the visual inspection of very small ducts and cavities, up to a 6-mm diameter. The system has been designed, implemented, and tested in a challenging industrial scenario and in strict collaboration with an avionic industry partner. The inspected objects are metallic gearboxes eventually presenting different residuals (e.g., sand, machining swarfs, and metallic dust) inside the oil ducts. The automatic system is actuated by a robotic arm that moves the endoscope with a microcamera inside the gearbox duct, while a deep-learning-based spatio-temporal image analysis module detects, classifies, and localizes defects in real time. Feedback is given to the robotic arm in order to move or extract the endoscope given the detected anomalies. Evaluation provides a detection rate of nearly $98$ % given different tests with different types of residuals and duct structures.

[1]  Taro Nakamura,et al.  Development of a peristaltic crawling inspection robot for 1-inch gas pipes with continuous elbows , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  Darwin G. Caldwell,et al.  Smart devices in robot-assisted laser microsurgery: Towards ubiquitous tele-cooperation , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[3]  Carlos Balaguer,et al.  Robot-aided tunnel inspection and maintenance system by vision and proximity sensor integration , 2011 .

[4]  Jürgen Schmidhuber,et al.  Learning to forget: continual prediction with LSTM , 1999 .

[5]  Navdeep Jaitly,et al.  Hybrid speech recognition with Deep Bidirectional LSTM , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.

[6]  Alberto Jardón,et al.  Past, present and future of robotic tunnel inspection , 2015 .

[7]  Kaspar Althoefer,et al.  State of the art in sensor technologies for sewer inspection , 2002 .

[8]  Osama Moselhi,et al.  Classification of Defects in Sewer Pipes Using Neural Networks , 2000 .

[9]  Nitish Srivastava,et al.  Unsupervised Learning of Video Representations using LSTMs , 2015, ICML.

[10]  Nabil Simaan,et al.  Snake-Like Units Using Flexible Backbones and Actuation Redundancy for Enhanced Miniaturization , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[11]  Koichi Suzumori,et al.  Micro inspection robot for 1-in pipes , 1999 .

[12]  Johan Schalkwyk,et al.  Learning acoustic frame labeling for speech recognition with recurrent neural networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Russell H. Taylor,et al.  High Dexterity Snake-Like Robotic Slaves for Minimally Invasive Telesurgery of the Upper Airway , 2004, MICCAI.

[14]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[15]  Michael R. Zinn,et al.  A combined modal-joint space control approach for continuum manipulators , 2014, Adv. Robotics.

[16]  Keiji Sasaki,et al.  Development of remote microsurgery robot and new surgical procedure for deep and narrow space , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[17]  Adrien Bartoli,et al.  Robust, Real-Time, Dense and Deformable 3D Organ Tracking in Laparoscopic Videos , 2016, MICCAI.

[18]  Ming-Der Yang,et al.  Segmenting ideal morphologies of sewer pipe defects on CCTV images for automated diagnosis , 2009, Expert Syst. Appl..

[19]  Daniel Mirota,et al.  Robust motion estimation and structure recovery from endoscopic image sequences with an Adaptive Scale Kernel Consensus estimator , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[21]  Edoardo Sinibaldi,et al.  The First Interlaced Continuum Robot, Devised to Intrinsically Follow the Leader , 2016, PloS one.

[22]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[25]  A. Ahrary,et al.  A New Approach to the Sewer Pipe Inspection: Fully Autonomous Mobile Robot "KANTARO" , 2006, IECON 2006 - 32nd Annual Conference on IEEE Industrial Electronics.

[26]  Christopher Joseph Pal,et al.  Describing Videos by Exploiting Temporal Structure , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[27]  Adrien Bartoli,et al.  Towards Live Monocular 3D Laparoscopy Using Shading and Specularity Information , 2012, IPCAI.

[28]  Ioannis K. Kaliakatsos,et al.  Microrobots for minimally invasive medicine. , 2010, Annual review of biomedical engineering.

[29]  Byung-Ju Yi,et al.  Design of a spring backbone micro endoscope , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[30]  Sm Balaji,et al.  Oral and maxillofacial surgery in India , 2007 .

[31]  Gerard Lacey,et al.  Automatic Segmentation and Inpainting of Specular Highlights for Endoscopic Imaging , 2010, EURASIP J. Image Video Process..

[32]  Tim Lüth,et al.  Quantitative measurements of soft tissue structures using image-features in navigated endoscopy , 2016, 2016 IEEE 5th Global Conference on Consumer Electronics.

[33]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Matthew J. Hausknecht,et al.  Beyond short snippets: Deep networks for video classification , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Nabil Simaan,et al.  Compliant Motion Control for Multisegment Continuum Robots With Actuation Force Sensing , 2014, IEEE Transactions on Robotics.

[36]  Shivprakash Iyer,et al.  A robust approach for automatic detection and segmentation of cracks in underground pipeline images , 2005, Image Vis. Comput..

[37]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.