R-JaunLab: Automatic Multi-Class Recognition of Jaundice on Photos of Subjects with Region Annotation Networks

Jaundice occurs as a symptom of various diseases, such as hepatitis, the liver cancer, gallbladder or pancreas. Therefore, clinical measurement with special equipment is a common method that is used to identify the total serum bilirubin level in patients. Fully automated multi-class recognition of jaundice combines two key issues: (1) the critical difficulties in multi-class recognition of jaundice approaches contrasting with the binary class and (2) the subtle difficulties in multi-class recognition of jaundice represent extensive individuals variability of high-resolution photos of subjects, huge coherency between healthy controls and occult jaundice, as well as broadly inhomogeneous color distribution. We introduce a novel approach for multi-class recognition of jaundice to detect occult jaundice, obvious jaundice and healthy controls. First, region annotation network is developed and trained to propose eye candidates. Subsequently, an efficient jaundice recognizer is proposed to learn similarities, context, localization features and globalization characteristics on photos of subjects. Finally, both networks are unified by using shared convolutional layer. Evaluation of the structured model in a comparative study resulted in a significant performance boost (categorical accuracy for mean 91.38%) over the independent human observer. Our work was exceeded against the state-of-the-art convolutional neural network (96.85% and 90.06% for training and validation subset, respectively) and showed a remarkable categorical result for mean 95.33% on testing subset. The proposed network makes a performance better than physicians. This work demonstrates the strength of our proposal to help bringing an efficient tool for multi-class recognition of jaundice into clinical practice.

[1]  Yann LeCun,et al.  Learning to Linearize Under Uncertainty , 2015, NIPS.

[2]  Munish Kumar,et al.  Content-based image retrieval system using ORB and SIFT features , 2018, Neural Computing and Applications.

[3]  Jae Y. Shin,et al.  Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? , 2016, IEEE transactions on medical imaging.

[4]  Roger Jones,et al.  Oxford Textbook of Primary Medical Care , 2004 .

[5]  Dinggang Shen,et al.  Scalable High Performance Image Registration Framework by Unsupervised Deep Feature Representations Learning , 2017, Deep Learning for Medical Image Analysis.

[6]  Hamed Golmohammadi,et al.  Easy Diagnosis of Jaundice: A Smartphone-Based Nanosensor Bioplatform Using Photoluminescent Bacterial Nanopaper for Point-of-Care Diagnosis of Hyperbilirubinemia. , 2019, ACS sensors.

[7]  C. Chambers,et al.  Intense jaundice in an adolescent. An unusual presentation of infectious mononucleosis. , 1986, Journal of adolescent health care : official publication of the Society for Adolescent Medicine.

[8]  Mohammadhossein Khorraminejad-Shirazi,et al.  Detection of Neonatal Jaundice by Using an Android OS-Based Smartphone Application , 2019, Iranian Journal of Pediatrics.

[9]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Licheng Jiao,et al.  Eye detection under varying illumination using the retinex theory , 2013, Neurocomputing.

[11]  Ronald M. Summers,et al.  Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique , 2016 .

[12]  Shuhei Nomura,et al.  Automatic anatomical classification of esophagogastroduodenoscopy images using deep convolutional neural networks , 2018, Scientific Reports.

[13]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[14]  Yuanjie Zheng,et al.  Breast Cancer Multi-classification from Histopathological Images with Structured Deep Learning Model , 2017, Scientific Reports.

[15]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Elena Marchiori,et al.  Location Sensitive Deep Convolutional Neural Networks for Segmentation of White Matter Hyperintensities , 2016, Scientific Reports.

[17]  Luiz Eduardo Soares de Oliveira,et al.  Breast cancer histopathological image classification using Convolutional Neural Networks , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[18]  Amod Kumar,et al.  Non-invasive Jaundice Detection using Machine Vision , 2013 .

[19]  L. Ben-Porat,et al.  Jaundice Predicts Advanced Disease and Early Mortality in Patients With Gallbladder Cancer , 2004, Annals of Surgical Oncology.

[20]  Gulshan Kumar,et al.  A Survey of Deep Learning and Its Applications: A New Paradigm to Machine Learning , 2019, Archives of Computational Methods in Engineering.

[21]  James M Winger,et al.  Diagnostic approach to the patient with jaundice. , 2011, Primary care.

[22]  R. Knill-Jones,et al.  Use of Sequential Bayesian Model in Diagnosis of Jaundice by Computer , 1973, British medical journal.

[23]  P. Myles,et al.  Using the Bland-Altman method to measure agreement with repeated measures. , 2007, British journal of anaesthesia.

[24]  S. Roche,et al.  Jaundice in the adult patient. , 2004, American family physician.

[25]  Munish Kumar,et al.  2D Object Recognition Techniques: State-of-the-Art Work , 2020 .

[26]  Keshav Pingali,et al.  Stochastic gradient descent on GPUs , 2015, GPGPU@PPoPP.

[27]  Thomas Brox,et al.  Discriminative Unsupervised Feature Learning with Convolutional Neural Networks , 2014, NIPS.

[28]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[29]  Sankhadip Saha,et al.  Classifier fusion for liver function test based Indian jaundice classification , 2015, 2015 International Conference on Man and Machine Interfacing (MAMI).

[30]  Michael Blum,et al.  High-Throughput Classification of Radiographs Using Deep Convolutional Neural Networks , 2016, Journal of Digital Imaging.

[31]  M. J. Maisels,et al.  Managing the jaundiced newborn: a persistent challenge , 2015, Canadian Medical Association Journal.

[32]  Heung-Il Suk,et al.  Deep Learning in Medical Image Analysis. , 2017, Annual review of biomedical engineering.

[33]  A. Anand,et al.  Jaundice in malaria , 2005, Journal of gastroenterology and hepatology.

[34]  Xuening Zhang,et al.  The Application of Intravoxel Incoherent Motion Diffusion-Weighted Imaging in the Diagnosis of Hilar Obstructive Jaundice , 2019, Journal of computer assisted tomography.

[35]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[36]  Hoo-Chang Hoo-Chang Shin Shin,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, Ieee Transactions on Medical Imaging.

[37]  D. Altman,et al.  Measuring agreement in method comparison studies , 1999, Statistical methods in medical research.

[38]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[39]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[42]  Hao Chen,et al.  Mitosis Detection in Breast Cancer Histology Images via Deep Cascaded Networks , 2016, AAAI.

[43]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[45]  Olga V. Demler,et al.  Clinical calibration of DSM‐IV diagnoses in the World Mental Health (WMH) version of the World Health Organization (WHO) Composite International Diagnostic Interview (WMH‐CIDI) , 2004, International journal of methods in psychiatric research.

[46]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[47]  A R Midgley,et al.  Principles for the assessment of the reliability of radioimmunoassay methods (precision, accuracy, sensitivity, specificity). , 1969, Acta endocrinologica. Supplementum.

[48]  Mark D. McDonnell,et al.  Understanding Data Augmentation for Classification: When to Warp? , 2016, 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[49]  Shiqian Ma,et al.  Highly accurate model for prediction of lung nodule malignancy with CT scans , 2018, Scientific Reports.

[50]  R. Kozarek,et al.  Nonoperative versus operative treatment of obstructive jaundice in pancreatic cancer: cost and survival analysis. , 1988, The American journal of gastroenterology.

[51]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[52]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[53]  Seong-Whan Lee,et al.  Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis , 2014, NeuroImage.

[54]  Christoph Meinel,et al.  Deep Learning for Medical Image Analysis , 2018, Journal of Pathology Informatics.

[55]  E. Marchiori,et al.  The role of percutaneous transhepatic biliary biopsy in the diagnosis of patients with obstructive jaundice: an initial experience , 2019, Radiologia brasileira.

[56]  Yu Meng,et al.  An Effective CNN Method for Fully Automated Segmenting Subcutaneous and Visceral Adipose Tissue on CT Scans , 2019, Annals of Biomedical Engineering.

[57]  H. Chan,et al.  The role of lamivudine and predictors of mortality in severe flare‐up of chronic hepatitis B with jaundice , 2002, Journal of viral hepatitis.

[58]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[59]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  G. Mortimore,et al.  Right hypochondrial pain leading to diagnosis of cholestatic jaundice and cholecystitis: a review and case study , 2019, Gastrointestinal Nursing.

[61]  Nima Tajbakhsh,et al.  Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? , 2016, IEEE Transactions on Medical Imaging.

[62]  Sriparna Saha,et al.  Graph-Based Hub Gene Selection Technique Using Protein Interaction Information: Application to Sample Classification , 2019, IEEE Journal of Biomedical and Health Informatics.

[63]  Debasis Bhattacharyya,et al.  A Novel Whole Spectrum-Based Non-Invasive Screening Device for Neonatal Hyperbilirubinemia , 2019, IEEE Journal of Biomedical and Health Informatics.

[64]  Aryuanto Soetedjo.  Eye Detection Based-on Color and Shape Features , 2012 .

[65]  Marios Anthimopoulos,et al.  Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network , 2016, IEEE Transactions on Medical Imaging.

[66]  K. Labori,et al.  Diagnostic Approach to the Patient with Jaundice following Trauma , 2004, Scandinavian journal of surgery : SJS : official organ for the Finnish Surgical Society and the Scandinavian Surgical Society.

[67]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[68]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  J. Landers,et al.  Simple, Reagentless Quantification of Total Bilirubin in Blood Via Microfluidic Phototreatment and Image Analysis. , 2017, Analytical chemistry.

[70]  Dayong Wang,et al.  Deep Learning for Identifying Metastatic Breast Cancer , 2016, ArXiv.

[71]  A. R. Ruzairi,et al.  An overview on jaundice measurement and application in biomedical: The potential of non-invasive method , 2015, 2015 2nd International Conference on Biomedical Engineering (ICoBE).

[72]  Anil A. Bharath,et al.  On denoising autoencoders trained to minimise binary cross-entropy , 2017, ArXiv.

[73]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[74]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[75]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[76]  Xiaomei Quan,et al.  Survey: Functional Module Detection from Protein-Protein Interaction Networks , 2014, IEEE Transactions on Knowledge and Data Engineering.

[77]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[78]  J. Clune,et al.  How Evolution Learns to Generalise: Principles of under-fitting, over-fitting and induction in the evolution of developmental organisation , 2015, 1508.06854.

[79]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[80]  L. Schieve,et al.  Neonatal jaundice in association with autism spectrum disorder and developmental disorder , 2019, Journal of Perinatology.

[81]  Munish Kumar,et al.  A comprehensive survey on the biometric recognition systems based on physiological and behavioral modalities , 2020, Expert Syst. Appl..

[82]  Firat Hardalaç,et al.  Neonatal Jaundice Detection System , 2016, Journal of Medical Systems.

[83]  David R. Myers,et al.  Smartphone app for non-invasive detection of anemia using only patient-sourced photos , 2018, Nature Communications.

[84]  Synho Do,et al.  How much data is needed to train a medical image deep learning system to achieve necessary high accuracy , 2015, 1511.06348.

[85]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[86]  B. van Ginneken,et al.  Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis , 2016, Scientific Reports.

[87]  Munish Kumar,et al.  A healthcare monitoring system using random forest and internet of things (IoT) , 2019, Multimedia Tools and Applications.

[88]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[89]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[90]  Dinggang Shen,et al.  Scalable High-Performance Image Registration Framework by Unsupervised Deep Feature Representations Learning , 2016, IEEE Transactions on Biomedical Engineering.

[91]  C. J. Watson,et al.  Antecedent jaundice in cirrhosis of the liver. , 1947, Archives of internal medicine.

[92]  Hui Sun,et al.  Urine Metabolomics Analysis for Biomarker Discovery and Detection of Jaundice Syndrome in Patients With Liver Disease* , 2012, Molecular & Cellular Proteomics.

[93]  Arakua Welbeck,et al.  Segmentation of the Proximal Femur from MR Images using Deep Convolutional Neural Networks , 2017, Scientific Reports.

[94]  Munish Kumar,et al.  2D-human face recognition using SIFT and SURF descriptors of face’s feature regions , 2020, The Visual Computer.