Deep Regression via Multi-Channel Multi-Modal Learning for Pneumonia Screening

Pneumonia screening is one of the most crucial steps in the pneumonia diagnosing system, which can improve the work efficiency of the radiologists and prevent delayed treatments. In this paper, we propose a deep regression framework for automatic pneumonia screening, which jointly learns the multi-channel images and multi-modal information (i.e., clinical chief complaints, age, and gender) to simulate the clinical pneumonia screening process. We demonstrate the advantages of the framework in three ways. First, visual features from multi-channel images (Lung Window Images, High Attenuation Images, Low Attenuation Images) can provide more visual features than single image channel, and improve the ability of screening pneumonia with severe diseases. Second, the proposed framework treats chest CT scans as short video frames and analyzes them by using Recurrent Convolutional Neural Network, which can automatically extract multiple image features from multi-channel image slices. Third, chief complaints and demographic information can provide valuable prior knowledge enhancing the features from images and further promote performance. The proposed framework has been extensively validated in 900 clinical cases. Compared to the baseline, the proposed framework improves the accuracy by 2.3% and significantly improves the sensitivity by 3.1%. To the best of our knowledge, we are the first to screen pneumonia using multi-channel images, multi-modal demographic and clinical information based on the large scale clinical raw dataset.

[1]  Zhou Li,et al.  Signal Analysis of Electrocardiogram and Statistical Evaluation of Myocardial Enzyme in the Diagnosis and Treatment of Patients With Pneumonia , 2019, IEEE Access.

[2]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Wei Liu,et al.  Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Shihui Ying,et al.  Multimodal Neuroimaging Feature Learning With Multimodal Stacked Deep Polynomial Networks for Diagnosis of Alzheimer's Disease , 2018, IEEE Journal of Biomedical and Health Informatics.

[5]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[6]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[7]  Dinggang Shen,et al.  Joint Classification and Regression via Deep Multi-Task Multi-Channel Learning for Alzheimer's Disease Diagnosis , 2019, IEEE Transactions on Biomedical Engineering.

[8]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Max A. Viergever,et al.  A Recurrent CNN for Automatic Detection and Classification of Coronary Artery Plaque and Stenosis in Coronary CT Angiography , 2018, IEEE Transactions on Medical Imaging.

[10]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[11]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[12]  Jianqing Li,et al.  Triple-Classification of Respiratory Sounds Using Optimized S-Transform and Deep Residual Networks , 2019, IEEE Access.

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Marleen de Bruijne,et al.  Machine learning approaches in medical image analysis: From detection to diagnosis , 2016, Medical Image Anal..

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Li Xiao,et al.  Alternating Diffusion Map Based Fusion of Multimodal Brain Connectivity Networks for IQ Prediction , 2018, IEEE Transactions on Biomedical Engineering.

[17]  Michalis A. Savelonas,et al.  Image Analysis Framework for Infection Monitoring , 2012, IEEE Transactions on Biomedical Engineering.

[18]  Li Yao,et al.  Learning to diagnose from scratch by exploiting dependencies among labels , 2017, ArXiv.

[19]  Andrew Zisserman,et al.  Convolutional Two-Stream Network Fusion for Video Action Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[21]  Ronald M. Summers,et al.  TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[23]  Ronald M. Summers,et al.  Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Kuan-Lun Tseng,et al.  Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Yu-Ping Wang,et al.  Deep Collaborative Learning With Application to the Study of Multimodal Brain Development , 2019, IEEE Transactions on Biomedical Engineering.

[26]  J. Carlin,et al.  Standardized interpretation of paediatric chest radiographs for the diagnosis of pneumonia in epidemiological studies. , 2005, Bulletin of the World Health Organization.

[27]  J. Dukart,et al.  Age Correction in Dementia – Matching to a Healthy Brain , 2011, PloS one.

[28]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[29]  Ronald M. Summers,et al.  Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks , 2018, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[30]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[31]  Andrew Y. Ng,et al.  CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning , 2017, ArXiv.

[32]  N. Deepika,et al.  Classification of Lobar Pneumonia by Two Different Classifiers in Lung CT Images , 2018, 2018 International Conference on Communication and Signal Processing (ICCSP).

[33]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[34]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[35]  Shyan-Ming Yuan,et al.  Cloud-Based Smart Dog Music Therapy and Pneumonia Detection System for Reducing the Difficulty of Caring for Patients With Dementia , 2020, IEEE Access.

[36]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[37]  Ronald M. Summers,et al.  ChestX-ray: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly Supervised Classification and Localization of Common Thorax Diseases , 2019, Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics.

[38]  Zhiyang He,et al.  Master clinical medical knowledge at certificated-doctor-level with deep learning model , 2018, Nature Communications.