Vision-based personalized Wireless Capsule Endoscopy for smart healthcare: Taxonomy, literature review, opportunities and challenges

Abstract Wireless Capsule Endoscopy (WCE) is a patient-friendly approach for digestive tract monitoring to support medical experts towards identifying any anomaly inside human’s Gastrointestinal (GI) tract. The automatic recognition of such type of abnormalities is essential for early diagnosis and time saving. To this end, several computer aided diagnosis (CAD) methods have been proposed in the literature for automatic abnormal region segmentation, summarization, classification, and personalization in WCE videos. In this work, we provide a detailed review of computer vision-based methods for WCE videos analysis. Firstly, all the major domains of WCE video analytics with their generic flow are identified. Secondly, we comprehensively review WCE video analysis methods and surveys with their pros and cons presented to date. In addition, this paper reviews several representative public datasets used for the performance assessment of WCE techniques and methods. Finally, the most important aspect of this survey is the identification of several research trends and open issues in different domains of WCE, with an emphasis placed on future research directions towards smarter healthcare and personalization.

[1]  N. Bourbakis,et al.  Wireless Capsule Endoscopy and Endoscopic Imaging: A Survey on Various Methodologies Presented , 2010, IEEE Engineering in Medicine and Biology Magazine.

[2]  Anant Suraj Vemuri,et al.  Survey of Computer Vision and Machine Learning in Gastrointestinal Endoscopy , 2019, ArXiv.

[3]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Kaveh Pahlavan,et al.  Design, Implementation, and Fundamental Limits of Image and RF Based Wireless Capsule Endoscopy Hybrid Localization , 2016, IEEE Transactions on Mobile Computing.

[5]  Sung Wook Baik,et al.  Visual saliency models for summarization of diagnostic hysteroscopy videos in healthcare systems , 2016, SpringerPlus.

[6]  Khan Muhammad,et al.  Random Forest with Self-Paced Bootstrap Learning in Lung Cancer Prognosis , 2020, ACM Trans. Multim. Comput. Commun. Appl..

[7]  Francisco Herrera,et al.  Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI , 2020, Inf. Fusion.

[8]  M. Maher Ben Ismail,et al.  Multiple bleeding detection in wireless capsule endoscopy , 2018, Signal, Image and Video Processing.

[9]  Sung Wook Baik,et al.  Cloud-Assisted Multiview Video Summarization Using CNN and Bidirectional LSTM , 2020, IEEE Transactions on Industrial Informatics.

[10]  Sung Wook Baik,et al.  Intelligent Baby Behavior Monitoring using Embedded Vision in IoT for Smart Healthcare Centers , 2019, Journal of Artificial Intelligence and Systems.

[11]  Javier Del Ser,et al.  Deep Learning for Multigrade Brain Tumor Classification in Smart Healthcare Systems: A Prospective Survey , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Mahdi Alizadeh,et al.  Feature Based Framework to Detect Diseases, Tumor, and Bleeding in Wireless Capsule Endoscopy , 2018, ArXiv.

[13]  Sung Wook Baik,et al.  Violence Detection Using Spatiotemporal Features with 3D Convolutional Neural Network , 2019, Sensors.

[14]  Mehmet Rasit Yuce,et al.  A Navigation and Pressure Monitoring System Toward Autonomous Wireless Capsule Endoscopy , 2020, IEEE Sensors Journal.

[15]  Sung Wook Baik,et al.  Medical Image Retrieval with Compact Binary Codes Generated in Frequency Domain Using Highly Reactive Convolutional Features , 2018, Journal of Medical Systems.

[16]  Ali Ghaffari,et al.  Image Based High-Level Control System Design for Steering and Controlling of an Active Capsule Endoscope , 2018, J. Intell. Robotic Syst..

[17]  Sung Wook Baik,et al.  Dual-Level Security based Cyclic18 Steganographic Method and its Application for Secure Transmission of Keyframes during Wireless Capsule Endoscopy , 2016, Journal of Medical Systems.

[18]  D. Iakovidis,et al.  Software for enhanced video capsule endoscopy: challenges for essential progress , 2015, Nature Reviews Gastroenterology &Hepatology.

[19]  Wenyao Xu,et al.  Smart Insole: A Wearable Sensor Device for Unobtrusive Gait Monitoring in Daily Life , 2016, IEEE Transactions on Industrial Informatics.

[20]  Sung Wook Baik,et al.  CNN-based anti-spoofing two-tier multi-factor authentication system , 2019, Pattern Recognit. Lett..

[21]  K. Koike,et al.  Automatic detection and classification of protruding lesions in wireless capsule endoscopy images based on a deep convolutional neural network. , 2020, Gastrointestinal endoscopy.

[22]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Pedro M. Vieira,et al.  Automatic Detection of Small Bowel Tumors in Wireless Capsule Endoscopy Images Using Ensemble Learning. , 2019, Medical physics.

[24]  Sung Wook Baik,et al.  Endoscopic Image Classification and Retrieval using Clustered Convolutional Features , 2017, Journal of Medical Systems.

[25]  Jiebo Luo,et al.  Deep sparse feature selection for computer aided endoscopy diagnosis , 2015, Pattern Recognit..

[26]  Cuntai Guan,et al.  A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Nikolaos G. Bourbakis,et al.  Detection of Small Bowel Polyps and Ulcers in Wireless Capsule Endoscopy Videos , 2011, IEEE Transactions on Biomedical Engineering.

[28]  K. Koike,et al.  Artificial intelligence using a convolutional neural network for automatic detection of small‐bowel angioectasia in capsule endoscopy images , 2020, Digestive endoscopy : official journal of the Japan Gastroenterological Endoscopy Society.

[29]  Sung Wook Baik,et al.  Efficient CNN based summarization of surveillance videos for resource-constrained devices , 2020, Pattern Recognit. Lett..

[30]  Saraju P. Mohanty,et al.  Smart-Log: A Deep-Learning Based Automated Nutrition Monitoring System in the IoT , 2018, IEEE Transactions on Consumer Electronics.

[31]  Byeong-Seok Shin,et al.  Reducing redundancy in wireless capsule endoscopy videos , 2013, Comput. Biol. Medicine.

[32]  Jun Zhong,et al.  Towards unsupervised physical activity recognition using smartphone accelerometers , 2016, Multimedia Tools and Applications.

[33]  Mohamed El Ansari,et al.  Computer-aided diagnosis system for colon abnormalities detection in wireless capsule endoscopy images , 2017, Multimedia Tools and Applications.

[34]  Mark Fisher,et al.  Colour Image Analysis of Wireless Capsule Endoscopy Video: A Review , 2013 .

[35]  Sung Wook Baik,et al.  Efficient Deep CNN-Based Fire Detection and Localization in Video Surveillance Applications , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[36]  V. B. Surya Prasath Polyp Detection and Segmentation from Video Capsule Endoscopy: A Review , 2016, J. Imaging.

[37]  Ebrahim Ghafar-Zadeh,et al.  Smart Cell Culture Monitoring and Drug Test Platform Using CMOS Capacitive Sensor Array , 2019, IEEE Transactions on Biomedical Engineering.

[38]  Mohamed El Ansari,et al.  Multi-scale completed local binary patterns for ulcer detection in wireless capsule endoscopy images , 2018, Multimedia Tools and Applications.

[39]  Zhihan Lv,et al.  Secure video summarization framework for personalized wireless capsule endoscopy , 2017, Pervasive Mob. Comput..

[40]  Jaime Lloret,et al.  Robust Image Hashing Based Efficient Authentication for Smart Industrial Environment , 2019, IEEE Transactions on Industrial Informatics.

[41]  Sung Wook Baik,et al.  Intelligent Embedded Vision for Summarization of Multiview Videos in IIoT , 2020, IEEE Transactions on Industrial Informatics.

[42]  Guang-Zhong Yang,et al.  Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery , 2017, ArXiv.

[43]  Guozheng Yan,et al.  Design and Testing of a Motor-Based Capsule Robot Powered by Wireless Power Transmission , 2016, IEEE/ASME Transactions on Mechatronics.

[44]  Klaus Schöffmann,et al.  Content-based processing and analysis of endoscopic images and videos: A survey , 2017, Multimedia Tools and Applications.

[45]  Sung Wook Baik,et al.  Activity Recognition Using Temporal Optical Flow Convolutional Features and Multilayer LSTM , 2019, IEEE Transactions on Industrial Electronics.

[46]  Ahnaf Rashik Hassan,et al.  Computer-aided gastrointestinal hemorrhage detection in wireless capsule endoscopy videos , 2015, Comput. Methods Programs Biomed..

[47]  Khan A. Wahid,et al.  Automated Adaptive Brightness in Wireless Capsule Endoscopy Using Image Segmentation and Sigmoid Function , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[48]  Jean-Michel Redouté,et al.  A Locomotion Control Platform With Dynamic Electromagnetic Field for Active Capsule Endoscopy , 2018, IEEE Journal of Translational Engineering in Health and Medicine.

[49]  Chong-Wah Ngo,et al.  Automatic Hookworm Detection in Wireless Capsule Endoscopy Images , 2016, IEEE Transactions on Medical Imaging.

[50]  Guozheng Yan,et al.  Bleeding Detection in Wireless Capsule Endoscopy Based on Probabilistic Neural Network , 2011, Journal of Medical Systems.

[51]  Susmita Sur-Kolay,et al.  Wearable Medical Sensor-Based System Design: A Survey , 2017, IEEE Transactions on Multi-Scale Computing Systems.

[52]  Arianna Menciassi,et al.  An Innovative Wireless Endoscopic Capsule With Spherical Shape , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[53]  Sung Wook Baik,et al.  A novel magic LSB substitution method (M-LSB-SM) using multi-level encryption and achromatic component of an image , 2015, Multimedia Tools and Applications.

[54]  Max Q.-H. Meng,et al.  Discriminative Joint-Feature Topic Model With Dual Constraints for WCE Classification , 2018, IEEE Transactions on Cybernetics.

[55]  Guang-Zhong Yang,et al.  Online tracking and retargeting with applications to optical biopsy in gastrointestinal endoscopic examinations , 2016, Medical Image Anal..

[56]  Khan Muhammad,et al.  DeepReS: A Deep Learning-Based Video Summarization Strategy for Resource-Constrained Industrial Surveillance Scenarios , 2020, IEEE Transactions on Industrial Informatics.

[57]  Jeongkyu Lee,et al.  A Review of Machine-Vision-Based Analysis of Wireless Capsule Endoscopy Video , 2012, Diagnostic and therapeutic endoscopy.

[58]  Sung Wook Baik,et al.  Energy-Efficient Deep CNN for Smoke Detection in Foggy IoT Environment , 2019, IEEE Internet of Things Journal.

[59]  Sung Wook Baik,et al.  Secure Surveillance Framework for IoT Systems Using Probabilistic Image Encryption , 2018, IEEE Transactions on Industrial Informatics.

[60]  Shidan Cheng,et al.  Magnetically Guided Capsule Endoscopy in Pediatric Patients with Abdominal Pain , 2019, Gastroenterology research and practice.

[61]  Sung Wook Baik,et al.  Movie scene segmentation using object detection and set theory , 2019, Int. J. Distributed Sens. Networks.

[62]  K. Koike,et al.  Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network. , 2019, Gastrointestinal endoscopy.

[63]  Max Q.-H. Meng,et al.  WCE Abnormality Detection Based on Saliency and Adaptive Locality-Constrained Linear Coding , 2017, IEEE Transactions on Automation Science and Engineering.

[64]  Max Q.-H. Meng,et al.  Tumor Recognition in Wireless Capsule Endoscopy Images Using Textural Features and SVM-Based Feature Selection , 2012, IEEE Transactions on Information Technology in Biomedicine.

[65]  Dimitris K. Iakovidis,et al.  Automatic lesion detection in capsule endoscopy based on color saliency: closer to an essential adjunct for reviewing software. , 2014, Gastrointestinal endoscopy.

[66]  Sung Wook Baik,et al.  Mobile-Cloud Assisted Video Summarization Framework for Efficient Management of Remote Sensing Data Generated by Wireless Capsule Sensors , 2014, Sensors.

[67]  Panos Liatsis,et al.  Application of Convolutional Neural Networks for Automated Ulcer Detection in Wireless Capsule Endoscopy Images , 2019, Sensors.

[68]  Sung Wook Baik,et al.  Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features , 2018, IEEE Access.

[69]  Seyed Shahim Vedaei,et al.  A Fluorescence-Based Wireless Capsule Endoscopy System for Detecting Colorectal Cancer , 2020, Cancers.

[70]  Niraj K. Jha,et al.  Smart, Secure, Yet Energy-Efficient, Internet-of-Things Sensors , 2018, IEEE Transactions on Multi-Scale Computing Systems.

[71]  Youngbae Hwang,et al.  3D reconstruction of small bowel lesions using stereo camera-based capsule endoscopy , 2020, Scientific Reports.

[72]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[73]  Md. Rubel Basar,et al.  Ingestible Wireless Capsule Technology: A Review of Development and Future Indication , 2012 .

[74]  Ramesh Jain,et al.  Hookworm Detection in Wireless Capsule Endoscopy Images With Deep Learning , 2018, IEEE Transactions on Image Processing.

[75]  Lihua Li,et al.  Computer-aided detection of small intestinal ulcer and erosion in wireless capsule endoscopy images , 2018, Physics in medicine and biology.

[76]  Sung Wook Baik,et al.  Multi-grade brain tumor classification using deep CNN with extensive data augmentation , 2019, J. Comput. Sci..

[77]  Mohamed Elhoseny,et al.  Efficient Fire Detection for Uncertain Surveillance Environment , 2019, IEEE Transactions on Industrial Informatics.

[78]  Nasser Kehtarnavaz,et al.  Improving Human Action Recognition Using Fusion of Depth Camera and Inertial Sensors , 2015, IEEE Transactions on Human-Machine Systems.

[79]  Nilanjan Dey,et al.  Wireless Capsule Gastrointestinal Endoscopy: Direction-of-Arrival Estimation Based Localization Survey , 2017, IEEE Reviews in Biomedical Engineering.

[80]  Reza Aghaeizadeh Zoroofi,et al.  Automatic reduction of wireless capsule endoscopy reviewing time based on factorization analysis , 2020, Biomed. Signal Process. Control..

[81]  Christian Biemann,et al.  What do we need to build explainable AI systems for the medical domain? , 2017, ArXiv.

[82]  Soo Young Shin,et al.  A survey on contemporary computer-aided tumor, polyp, and ulcer detection methods in wireless capsule endoscopy imaging , 2019, Comput. Medical Imaging Graph..

[83]  Andreas Uhl,et al.  Survey on computer aided decision support for diagnosis of celiac disease , 2015, Comput. Biol. Medicine.

[84]  Max Q.-H. Meng,et al.  Improved Bag of Feature for Automatic Polyp Detection in Wireless Capsule Endoscopy Images , 2016, IEEE Transactions on Automation Science and Engineering.

[85]  Wei Hu,et al.  A CMOS Smart Temperature Sensor With Single-Point Calibration Method for Clinical Use , 2016, IEEE Transactions on Circuits and Systems II: Express Briefs.

[86]  Max Q.-H. Meng,et al.  Bleeding Frame and Region Detection in the Wireless Capsule Endoscopy Video , 2016, IEEE Journal of Biomedical and Health Informatics.

[87]  Dimitrios K. Iakovidis,et al.  Reduction of capsule endoscopy reading times by unsupervised image mining , 2010, Comput. Medical Imaging Graph..

[88]  Guang-Zhong Yang,et al.  Probabilistic Tracking of Affine-Invariant Anisotropic Regions , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[89]  Khan Muhammad,et al.  Hiding medical information in brain MR images without affecting accuracy of classifying pathological brain , 2019, Future Gener. Comput. Syst..

[90]  Faisal Mahmood,et al.  Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training , 2017, IEEE Transactions on Medical Imaging.

[91]  Atsushi Saito,et al.  Bleeding Classification of Enhanced Wireless Capsule Endoscopy Images using Deep Convolutional Neural Network , 2020, J. Inf. Sci. Eng..

[92]  Dimitris K. Iakovidis,et al.  Detecting and Locating Gastrointestinal Anomalies Using Deep Learning and Iterative Cluster Unification , 2018, IEEE Transactions on Medical Imaging.

[93]  Sung Wook Baik,et al.  DeepStar: Detecting Starring Characters in Movies , 2019, IEEE Access.

[94]  E. Redondo-Cerezo,et al.  Wireless capsule endoscopy: perspectives beyond gastrointestinal bleeding. , 2014, World journal of gastroenterology.

[95]  Nader Karimi,et al.  Segmentation of Bleeding Regions in Wireless Capsule Endoscopy for Detection of Informative Frames , 2018, Biomed. Signal Process. Control..

[96]  Yuxiang Xing,et al.  A systematic evaluation and optimization of automatic detection of ulcers in wireless capsule endoscopy on a large dataset using deep convolutional neural networks , 2019, Physics in medicine and biology.

[97]  Sung Wook Baik,et al.  Video summarization based tele-endoscopy: a service to efficiently manage visual data generated during wireless capsule endoscopy procedure , 2014, Journal of Medical Systems.

[98]  Lei Xing,et al.  Wireless Capsule Endoscopy: A New Tool for Cancer Screening in the Colon With Deep-Learning-Based Polyp Recognition , 2020, Proceedings of the IEEE.

[99]  Khan Muhammad,et al.  Cost-Effective Video Summarization Using Deep CNN With Hierarchical Weighted Fusion for IoT Surveillance Networks , 2020, IEEE Internet of Things Journal.

[100]  Hao Zhou,et al.  Enhanced Localization of Robotic Capsule Endoscopes Using Positron Emission Markers and Rigid-Body Transformation , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[101]  Guang-Zhong Yang,et al.  From Wearable Sensors to Smart Implants-–Toward Pervasive and Personalized Healthcare , 2015, IEEE Transactions on Biomedical Engineering.

[102]  Eyal Klang,et al.  Deep learning for wireless capsule endoscopy: a systematic review and meta-analysis. , 2020, Gastrointestinal endoscopy.

[103]  Sung Wook Baik,et al.  Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services , 2019, Inf. Sci..

[104]  Sung Wook Baik,et al.  Efficient visual attention driven framework for key frames extraction from hysteroscopy videos , 2017, Biomed. Signal Process. Control..

[105]  Max Q.-H. Meng,et al.  Saliency Based Ulcer Detection for Wireless Capsule Endoscopy Diagnosis , 2015, IEEE Transactions on Medical Imaging.

[106]  Shangbo Zhou,et al.  Deep Convolutional Neural Networks for WCE Abnormality Detection: CNN Architecture, Region Proposal and Transfer Learning , 2019, IEEE Access.

[107]  V. Konda,et al.  Wireless Capsule Endoscopy , 2020, Geriatric Gastroenterology.

[108]  Mubashir Husain Rehmani,et al.  A survey of feature extraction and fusion of deep learning for detection of abnormalities in video endoscopy of gastrointestinal-tract , 2019, Artificial Intelligence Review.

[109]  Paolo Bellavista,et al.  A privacy-preserving cryptosystem for IoT E-healthcare , 2020, Inf. Sci..

[110]  Sung Wook Baik,et al.  Personalized Movie Summarization Using Deep CNN-Assisted Facial Expression Recognition , 2019, Complex..

[111]  Dimitris K. Iakovidis,et al.  Video-based measurements for wireless capsule endoscope tracking , 2014 .

[112]  Huan Yang,et al.  Application of Structural Similarity Analysis of Visually Salient Areas and Hierarchical Clustering in the Screening of Similar Wireless Capsule Endoscopic Images , 2020, ArXiv.

[113]  Mingzhu Long,et al.  Adaptive Image Enhancement Based on Guide Image and Fraction-Power Transformation for Wireless Capsule Endoscopy , 2018, IEEE Transactions on Biomedical Circuits and Systems.