An automatic framework for endoscopic image restoration and enhancement

Despite its success in the field of minimally invasive surgery, endoscopy image analysis remains challenging due to limited image settings and control conditions. The low resolution and existence of large number of reflections in endoscopy images are the major problems in the automatic detection of objects. To address these issues, we presented a novel framework based on the convolutional neural networks. The proposed approach consists of three major parts. First, a deep learning (DL)-based image evaluation method is used to classify the input images into two groups, namely, specular highlights and weakly illuminated groups. Second, the specular highlight is detected using the DL-based method, and the reflected areas are recovered through a patch-based restoration operation. Lastly, gamma correction with optimized reflectance and illumination estimation is adopted to enhance the weakly illuminated images. The proposed method is compared against the existing ones, and the experimental results demonstrate that the former outperforms the latter in terms of subjective and objective assessments. This finding indicates that the proposed approach can serve as a potential tool for improving the quality of the endoscopy images used to examine internal body organs.

[1]  Berkman Sahiner,et al.  Deep learning in medical imaging and radiation therapy. , 2018, Medical physics.

[2]  Fernando Vilariño,et al.  Towards automatic polyp detection with a polyp appearance model , 2012, Pattern Recognit..

[3]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[4]  Han Lin,et al.  Review of Scene Text Detection and Recognition , 2020, Archives of Computational Methods in Engineering.

[5]  Max Q.-H. Meng,et al.  Wireless capsule endoscopy images enhancement via adaptive contrast diffusion , 2012, J. Vis. Commun. Image Represent..

[6]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[7]  Derek Nowrouzezahrai,et al.  Learning hatching for pen-and-ink illustration of surfaces , 2012, TOGS.

[8]  Chih-Hsien Hsia,et al.  A 3D Endoscopic Imaging System With Content-Adaptive Filtering and Hierarchical Similarity Analysis , 2016, IEEE Sensors Journal.

[9]  Michael Elad,et al.  A Variational Framework for Retinex , 2002, IS&T/SPIE Electronic Imaging.

[10]  Yi Li,et al.  Wireless capsule endoscopy images enhancement based on adaptive anisotropic diffusion , 2013, 2013 IEEE China Summit and International Conference on Signal and Information Processing.

[11]  Amir Hussain,et al.  Lip-Reading Driven Deep Learning Approach for Speech Enhancement , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[12]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[13]  Feifei Lee,et al.  A deep convolutional neural network architecture for interstitial lung disease pattern classification , 2020, Medical & Biological Engineering & Computing.

[14]  Othmane El Meslouhi,et al.  Automatic detection and inpainting of specular reflections for colposcopic images , 2011, Central European Journal of Computer Science.

[15]  Chen Wei,et al.  GLADNet: Low-Light Enhancement Network with Global Awareness , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[16]  Ausif Mahmood,et al.  Review of Deep Learning Algorithms and Architectures , 2019, IEEE Access.

[17]  Aymeric Histace,et al.  Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results From the MICCAI 2015 Endoscopic Vision Challenge , 2017, IEEE Transactions on Medical Imaging.

[18]  Aymeric Histace,et al.  Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer , 2014, International Journal of Computer Assisted Radiology and Surgery.

[19]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[20]  Khan A. Wahid,et al.  Image enhancement and space-variant color reproduction method for endoscopic images using adaptive sigmoid function , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[21]  Alicia Casals,et al.  Adaptive segmentation and mask-specific Sobolev inpainting of specular highlights for endoscopic images , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[22]  Inês Domingues,et al.  Computer Vision in Esophageal Cancer: A Literature Review , 2019, IEEE Access.

[23]  A. Lavecchia.  Deep learning in drug discovery: opportunities, challenges and future prospects. , 2019, Drug discovery today.

[24]  Farida Cheriet,et al.  Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images , 2007, Machine Vision and Applications.

[25]  Guodong Guo,et al.  A survey on deep learning based face recognition , 2019, Comput. Vis. Image Underst..

[26]  Tanveer F. Syeda-Mahmood,et al.  Building medical image classifiers with very limited data using segmentation networks , 2018, Medical Image Anal..

[27]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Hyun Sung Chang,et al.  Toward Specular Removal from Natural Images Based on Statistical Reflection Models , 2020, IEEE Transactions on Image Processing.

[29]  M. Ali Akber Dewan,et al.  A Dynamic Histogram Equalization for Image Contrast Enhancement , 2007, IEEE Transactions on Consumer Electronics.

[30]  Xiao-Ping Zhang,et al.  A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Chunxia Xiao,et al.  Specular Highlight Removal for Real‐world Images , 2019, Comput. Graph. Forum.

[32]  Bradley J. Erickson,et al.  Deep Learning in Radiology: Does One Size Fit All? , 2018, Journal of the American College of Radiology : JACR.

[33]  Jianhua Wu,et al.  MBLLEN: Low-Light Image/Video Enhancement Using CNNs , 2018, BMVC.

[34]  Michael Riegler,et al.  KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection , 2017, MMSys.

[35]  Faouzi Alaya Cheikh,et al.  Comparative study of endoscopic image enhancement techniques , 2015, 2015 Colour and Visual Computing Symposium (CVCS).

[36]  Khan Wahid,et al.  A color reproduction method with image enhancement for endoscopic images , 2014, 2nd Middle East Conference on Biomedical Engineering.

[37]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.

[38]  Khaled Shaalan,et al.  Speech Recognition Using Deep Neural Networks: A Systematic Review , 2019, IEEE Access.

[39]  Luc Soler,et al.  Evaluation of Endoscopic Image Enhancement for Feature Tracking: A New Validation Framework , 2013, AE-CAI.

[40]  Guo-Shiang Lin,et al.  A Specular Reflection Suppression Method for Endoscopic Images , 2016, 2016 IEEE Second International Conference on Multimedia Big Data (BigMM).

[41]  Hayit Greenspan,et al.  Automatic detection of specular reflections in uterine cervix images , 2006, SPIE Medical Imaging.

[42]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Xiangjian He,et al.  Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges , 2019, Journal of Digital Imaging.

[44]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[45]  Fatih Murat Porikli,et al.  LightenNet: A Convolutional Neural Network for weakly illuminated image enhancement , 2018, Pattern Recognit. Lett..

[46]  Ronald M. Summers,et al.  Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique , 2016 .