Towards realistic laparoscopic image generation using image-domain translation

Background and ObjectivesOver the last decade, Deep Learning (DL) has revolutionized data analysis in many areas, including medical imaging. However, there is a bottleneck in the advancement of DL in the surgery field, which can be seen in a shortage of large-scale data, which in turn may be attributed to the lack of a structured and standardized methodology for storing and analyzing surgical images in clinical centres. Furthermore, accurate annotations manually added are expensive and time consuming. A great help can come from the synthesis of artificial images; in this context, in the latest years, the use of Generative Adversarial Neural Networks (GANs) achieved promising results in obtaining photo-realistic images. MethodsIn this study, a method for Minimally Invasive Surgery (MIS) image synthesis is proposed. To this aim, the generative adversarial network pix2pix is trained to generate paired annotated MIS images by transforming rough segmentation of surgical instruments and tissues into realistic images. An additional regularization term was added to the original optimization problem, in order to enhance realism of surgical tools with respect to the background. Results Quantitative and qualitative (i.e., human-based) evaluations of generated images have been carried out in order to assess the effectiveness of the method. ConclusionsExperimental results show that the proposed method is actually able to translate MIS segmentations to realistic MIS images, which can in turn be used to augment existing data sets and help at overcoming the lack of useful images; this allows physicians and algorithms to take advantage from new annotated instances for their training.

[1]  Blake Hannaford,et al.  Real-time vision-based surgical tool segmentation with robot kinematics prior , 2018, 2018 International Symposium on Medical Robotics (ISMR).

[2]  Danail Stoyanov,et al.  Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy , 2019, International Journal of Computer Assisted Radiology and Surgery.

[3]  Giorgio Terracina,et al.  Biomedical Data Augmentation Using Generative Adversarial Neural Networks , 2017, ICANN.

[4]  D. Stoyanov,et al.  3-D Pose Estimation of Articulated Instruments in Robotic Minimally Invasive Surgery , 2018, IEEE Transactions on Medical Imaging.

[5]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[6]  Hongliang Ren,et al.  Real-Time Instrument Segmentation in Robotic Surgery Using Auxiliary Supervised Deep Adversarial Learning , 2019, IEEE Robotics and Automation Letters.

[7]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[8]  Elena De Momi,et al.  Dense soft tissue 3D reconstruction refined with super-pixel segmentation for robotic abdominal surgery , 2016, International Journal of Computer Assisted Radiology and Surgery.

[9]  Jaydeep H Palep,et al.  Robotic assisted minimally invasive surgery , 2009, Journal of Minimal Access Surgery.

[10]  Rüdiger Dillmann,et al.  LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognition , 2015, International Journal of Computer Assisted Radiology and Surgery.

[11]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[12]  Sébastien Ourselin,et al.  Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery , 2018, ArXiv.

[13]  Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks , 2018, Ieee Transactions on Medical Imaging.

[14]  Lena Maier-Hein,et al.  Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy , 2017, IEEE Transactions on Biomedical Engineering.

[15]  Debora Gil,et al.  Augmentation of Virtual Endoscopic Images with Intra-operative Data using Content-Nets , 2019 .

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[19]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Weiguo Chen,et al.  Learning from adversarial medical images for X-ray breast mass segmentation , 2019, Comput. Methods Programs Biomed..

[21]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[22]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[23]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[24]  Saad Nadeem,et al.  Augmenting Colonoscopy Using Extended and Directional CycleGAN for Lossy Image Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Masahiro Oda,et al.  Realistic endoscopic image generation method using virtual-to-real image-domain translation , 2019, Healthcare technology letters.

[26]  Shahrokh Valaee,et al.  Generalization of Deep Neural Networks for Chest Pathology Classification in X-Rays Using Generative Adversarial Networks , 2017, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Sebastian Bodenstedt,et al.  Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video , 2019, MICCAI.

[28]  Danail Stoyanov,et al.  Deep Learning Based Robotic Tool Detection and Articulation Estimation With Spatio-Temporal Layers , 2019, IEEE Robotics and Automation Letters.

[29]  Saeid Nahavandi,et al.  Surgical tool segmentation using a hybrid deep CNN-RNN auto encoder-decoder , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[30]  Cameron N. Riviere,et al.  Toward Improving Safety in Neurosurgery with an Active Handheld Instrument , 2018, Annals of Biomedical Engineering.

[31]  Nicolai Schoch,et al.  Surgical Data Science: Enabling Next-Generation Surgery , 2017, ArXiv.

[32]  Kang Ryoung Park,et al.  Conditional Generative Adversarial Network- Based Data Augmentation for Enhancement of Iris Recognition Accuracy , 2019, IEEE Access.

[33]  Hayit Greenspan,et al.  GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification , 2018, Neurocomputing.

[34]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[35]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[36]  Arianna Menciassi,et al.  Soft Robots in Surgery , 2017 .

[37]  Emanuele Frontoni,et al.  Inter-foetus Membrane Segmentation for TTTS Using Adversarial Networks , 2019, Annals of Biomedical Engineering.

[38]  Nassir Navab,et al.  Concurrent Segmentation and Localization for Tracking of Surgical Instruments , 2017, MICCAI.

[39]  Klaus H. Maier-Hein,et al.  Crowd-Algorithm Collaboration for Large-Scale Endoscopic Image Annotation with Confidence , 2016, MICCAI.

[40]  Franco Scarselli,et al.  Image generation by GAN and style transfer for agar plate image segmentation , 2019, Comput. Methods Programs Biomed..

[41]  Danna Zhou,et al.  d. , 1840, Microbial pathogenesis.