Applying Cross-Modality Data Processing for Infarction Learning in Medical Internet of Things

Cross-modality data processing is critical for the Internet-of-Things (IoT) deployment in healthcare. It can convert the innumerable raw day-to-day medical big data from massive IoT-based medical devices to diagnostic valuable data so that they can be feed to clinical routine. In this article, we propose a novel spatiotemporal two-streams generative adversarial network (SpGAN) as a cross-modality data processing approach to deploy the medical IoT in infarction learning. Our SpGAN remotely converts diagnostic valuable contrast-enhanced images (the “gold standard” for infarction learning, but it requires the injection of contrast agents) directly from raw nonenhanced cine MR images. This converting allows physicians to remotely perform infarction observation and analysis to break through the limitations of time and space by building a cloud computing platform of IoT-based MRI devices. Importantly, this converting offers a low-risk IoT-based manner to eliminate the potential fatal risk caused by contrast agent injection in the current infarction learning workflow. Specifically, SpGAN consists of: 1) a spatiotemporal two-stream framework as an encoding–decoding model to achieve data converting and 2) a spatiotemporal pyramid network enhances those features that are responsible to the infarction learning during encoding to improve decoding performance. Real IoT-based remote diagnosis experiments performed on 230 patients demonstrate that SpGAN provides high-quality converted images for infarction learning and promotes the in-depth application and deployment of IoT in the medical field.

[1]  Heye Zhang,et al.  PV-LVNet: Direct left ventricle multitype indices estimation from 2D echocardiograms of paired apical views with deep neural networks , 2019, Medical Image Anal..

[2]  Honggang Wang,et al.  A survey of big data research , 2015, IEEE Network.

[3]  Han Xiaojuan,et al.  Robust EEG-based Emotion Recognition using Multi-feature Joint Sparse Representation , 2020 .

[4]  Sotirios A. Tsaftaris,et al.  Medical Image Computing and Computer Assisted Intervention , 2017 .

[5]  W. Marsden.  I and J , 2012 .

[6]  Heye Zhang,et al.  Robust estimation of carotid artery wall motion using the elasticity‐based state‐space approach , 2017, Medical Image Anal..

[7]  Danna Zhou,et al.  d. , 1840, Microbial pathogenesis.

[8]  Heye Zhang,et al.  MuTGAN: Simultaneous Segmentation and Quantification of Myocardial Infarction Without Contrast Agents via Joint Adversarial Learning , 2018, MICCAI.

[9]  P. Alam.  ‘A’ , 2021, Composites Engineering: An A–Z Guide.

[10]  Ting-Zhu Huang,et al.  Image restoration using total variation with overlapping group sparsity , 2013, Inf. Sci..

[11]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[12]  Cheng Guo,et al.  Enabling Secure Cross-Modal Retrieval Over Encrypted Heterogeneous IoT Databases With Collective Matrix Factorization , 2020, IEEE Internet of Things Journal.

[13]  Lok Hin Lee,et al.  Automatic Determination of the Fetal Cardiac Cycle in Ultrasound Using Spatio-Temporal Neural Networks , 2020, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[14]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[15]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[16]  N. L. Zelyanskaya,et al.  Human-Oriented IoT-Based Interfaces for Multimodal Visual Analytics Systems , 2019 .

[17]  Thomas de Quincey.  [C] , 2000, The Works of Thomas De Quincey, Vol. 1: Writings, 1799–1820.

[18]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[19]  Yixue Hao,et al.  Label-less Learning for Emotion Cognition , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[20]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[21]  P. Alam,et al.  H , 1887, High Explosives, Propellants, Pyrotechnics.

[22]  C. Higgins,et al.  Delayed Contrast Enhancement on MR Images of Myocardium , 2011 .

[23]  Jessica M. Langer,et al.  Safe Use of Contrast Media: What the Radiologist Needs to Know. , 2015, Radiographics : a review publication of the Radiological Society of North America, Inc.

[24]  Nishant Ravikumar,et al.  Spatio-Temporal Multi-Task Learning for Cardiac MRI Left Ventricle Quantification , 2020, IEEE Journal of Biomedical and Health Informatics.

[25]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[26]  Miguel Ángel González Ballester,et al.  Combining Multi-Sequence and Synthetic Images for Improved Segmentation of Late Gadolinium Enhancement Cardiac MRI , 2019, STACOM@MICCAI.

[27]  Chi-Sheng Shih,et al.  Supporting Service Adaptation in Fault Tolerant Internet of Things , 2015, 2015 IEEE 8th International Conference on Service-Oriented Computing and Applications (SOCA).

[28]  Felix Lau,et al.  ScarGAN: Chained Generative Adversarial Networks to Simulate Pathological Tissue on Cardiovascular MR Scans , 2018, DLMIA/ML-CDS@MICCAI.

[29]  G. Fitzgerald,et al.  'I. , 2019, Australian journal of primary health.

[30]  Heye Zhang,et al.  Direct Detection of Pixel-Level Myocardial Infarction Areas via a Deep-Learning Algorithm , 2017, MICCAI.

[31]  Wei Li,et al.  Edge cognitive computing based smart healthcare system , 2018, Future Gener. Comput. Syst..

[32]  Synthetic LGE derived from cardiac T1 mapping for simultaneous assessment of focal and diffuse cardiac fibrosis , 2014, Journal of Cardiovascular Magnetic Resonance.

[33]  W. Hager,et al.  and s , 2019, Shallow Water Hydraulics.

[34]  Glenda C. Prkachin,et al.  The effects of orientation on detection and identification of facial expressions of emotion. , 2003, British journal of psychology.

[35]  Yu Bai,et al.  Multimodal Biometrics for Enhanced IoT Security , 2019, 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC).

[36]  Francesc Moreno-Noguer,et al.  GANimation: Anatomically-aware Facial Animation from a Single Image , 2018, ECCV.

[37]  Dong Liang,et al.  Motion Tracking of the Carotid Artery Wall From Ultrasound Image Sequences: a Nonlinear State-Space Approach , 2018, IEEE Transactions on Medical Imaging.

[38]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Amit P. Sheth,et al.  Internet of Things to Smart IoT Through Semantic, Cognitive, and Perceptual Computing , 2016, IEEE Intelligent Systems.

[40]  Limin Luo,et al.  K-Net: Integrate Left Ventricle Segmentation and Direct Quantification of Paired Echo Sequence , 2019, IEEE Transactions on Medical Imaging.

[41]  Hamid Sharif,et al.  Multimedia communications over cognitive radio networks for smart grid applications , 2013, IEEE Wireless Communications.

[42]  Huafeng Liu,et al.  Direct delineation of myocardial infarction without contrast agents using a joint motion feature learning architecture , 2018, Medical Image Anal..

[43]  Hodjat Hamidi,et al.  An approach to develop the smart health using Internet of Things and authentication based on biometric technology , 2019, Future Gener. Comput. Syst..

[44]  Feng Zhang,et al.  Ensemble Meteorological Cloud Classification Meets Internet of Dependable and Controllable Things , 2021, IEEE Internet of Things Journal.

[45]  Minghui Dai,et al.  Task Allocation With Unmanned Surface Vehicles in Smart Ocean IoT , 2020, IEEE Internet of Things Journal.

[46]  M. Shamim Hossain,et al.  An Audio-Visual Emotion Recognition System Using Deep Learning Fusion for a Cognitive Wireless Framework , 2019, IEEE Wireless Communications.

[47]  Min Chen,et al.  Label-less Learning for Traffic Control in an Edge Network , 2018, IEEE Network.

[48]  R. Sarpong,et al.  Bio-inspired synthesis of xishacorenes A, B, and C, and a new congener from fuscol† †Electronic supplementary information (ESI) available. See DOI: 10.1039/c9sc02572c , 2019, Chemical science.