OF-UMRN: Uncertainty-guided multitask regression network aided by optical flow for fully automated comprehensive analysis of carotid artery

Fully automated comprehensive analysis of carotid artery (localization of range of interest (ROI), direct quantitative measurement and segmentation of lumen diameter (CALD) and intima-media thickness (CIMT), and motion estimation of the carotid wall) is a reliable auxiliary diagnosis of cardiovascular diseases, which relieves physicians from laborious workloads. No work has achieved fully automated comprehensive analysis of carotid artery due to five intractable challenges: (1) The heavy reliance on experienced carotid physicians for the selection of ROI limits fully automated studies. (2) The weak structural information of intima-media thickness increases the difficulty of feature encoding. (3) The radial motion of the carotid wall results in the lack of discriminant features of boundaries. (4) Diseased carotid arteries lose many expression features. (5) Optimal weights of multitask regression are hard to tune manually. In this paper, we propose a novel uncertainty-guided multitask regression network aided by optical flow named OF-UMRN to solve the intractable challenges. The four modules and three innovations of the OF-UMRN take their responsibility. OF-UMRN takes localization and mapping of ROI as a pre-processing. It achieves direct quantitative measurement and segmentation by a multitask regression network. And we creatively model homoscedastic uncertainty to automated tune the weights of the two tasks optimally. The OF-UMRN adopts a bidirectional mechanism to encode the optical flow used to predict the carotid wall's motion fields. More importantly, we creatively propose a dual optimization module based on the co-promotion between segmentation and motion estimation to improve the performance of radially moving and diseased carotid arteries. Therefore, the OF-UMRN makes the most of the pathological relationship between multiple objects and co-promotion between various tasks. Extensive experiments on US sequences of 101 patients have demonstrated the superior performance of OF-UMRN on the fully automated comprehensive analysis of the carotid artery. Therefore the OF-UMRN has excellent potential in clinical disease diagnoses and assessments of the carotid artery.

[1]  Roberto Cipolla,et al.  Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[4]  T. Ryan,et al.  American society of echocardiography , 2000 .

[5]  Daniel Rueckert,et al.  Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences , 2018, MICCAI.

[6]  C. Rajasekaran,et al.  Convolutional Neural Network for Segmentation and Measurement of Intima Media Thickness , 2018, Journal of Medical Systems.

[7]  Heye Zhang,et al.  Direct Quantification for Coronary Artery Stenosis Using Multiview Learning , 2019, MICCAI.

[8]  G. Baltgaile.  Arterial wall dynamics , 2012 .

[9]  Yaping Tian,et al.  The correlation between serum lipid profile with carotid intima-media thickness and plaque , 2014, BMC Cardiovascular Disorders.

[10]  Heye Zhang,et al.  MuTGAN: Simultaneous Segmentation and Quantification of Myocardial Infarction Without Contrast Agents via Joint Adversarial Learning , 2018, MICCAI.

[11]  Michael J. Black,et al.  Video Segmentation via Object Flow , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Andrés Bueno-Crespo,et al.  Estimation of the Arterial Diameter in Ultrasound Images of the Common Carotid Artery , 2015, IWINAC.

[14]  Wufeng Xue,et al.  Full left ventricle quantification via deep multitask relationships learning , 2018, Medical Image Anal..

[15]  Christian Ledig,et al.  Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Christopher B. Kendall,et al.  Use of carotid ultrasound to identify subclinical vascular disease and evaluate cardiovascular disease risk: a consensus statement from the American Society of Echocardiography Carotid Intima-Media Thickness Task Force. Endorsed by the Society for Vascular Medicine. , 2008, Journal of the American Society of Echocardiography : official publication of the American Society of Echocardiography.

[17]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[19]  Qianjin Feng,et al.  Direct automated quantitative measurement of spine by cascade amplifier regression network with manifold regularization , 2019, Medical Image Anal..

[20]  P. Touboul,et al.  Factors of carotid arterial enlargement in a population aged 59 to 71 years: the EVA study. , 1996, Stroke.

[21]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[22]  Zoran Bursac,et al.  Common carotid artery wall thickness and external diameter as predictors of prevalent and incident cardiac events in a large population study , 2007, Cardiovascular ultrasound.

[23]  T. Naqvi,et al.  Carotid intima-media thickness and plaque in cardiovascular risk assessment. , 2014, JACC. Cardiovascular imaging.

[24]  Shuo Li,et al.  OF-MSRN: Optical Flow-Auxiliary Multi-Task Regression Network for Direct Quantitative Measurement, Segmentation and Motion Estimation , 2020, AAAI.

[25]  Jasjit S. Suri,et al.  Deep learning strategy for accurate carotid intima-media thickness measurement: An ultrasound study on Japanese diabetic cohort , 2018, Comput. Biol. Medicine.

[26]  Diederick E Grobbee,et al.  Common Carotid Intima-Media Thickness and Risk of Acute Myocardial Infarction: The Role of Lumen Diameter , 2005, Stroke.

[27]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[28]  Karl Rohr,et al.  Multi-channel Deep Transfer Learning for Nuclei Segmentation in Glioblastoma Cell Tissue Images , 2018, Bildverarbeitung für die Medizin.

[29]  Effat Soleimani,et al.  Carotid Artery Wall Motion Estimation from Consecutive Ultrasonic Images: Comparison between Block-Matching and Maximum-Gradient Algorithms , 2011, The journal of Tehran Heart Center.

[30]  Jasjit S. Suri,et al.  Deep learning fully convolution network for lumen characterization in diabetic patients using carotid ultrasound: a tool for stroke risk , 2018, Medical & Biological Engineering & Computing.

[31]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Zhichao Yin,et al.  GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Nima Tajbakhsh,et al.  Automating Carotid Intima-Media Thickness Video Interpretation with Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Enzo Grossi,et al.  Measurements of carotid intima-media thickness and of interadventitia common carotid diameter improve prediction of cardiovascular events: results of the IMPROVE (Carotid Intima Media Thickness [IMT] and IMT-Progression as Predictors of Vascular Events in a High Risk European Population) study. , 2012, Journal of the American College of Cardiology.

[35]  Roberto Cipolla,et al.  Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding , 2015, BMVC.

[36]  Klaus H. Maier-Hein,et al.  A Probabilistic U-Net for Segmentation of Ambiguous Images , 2018, NeurIPS.

[37]  Jan Kautz,et al.  PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Andrew Zisserman,et al.  Multi-task Self-Supervised Visual Learning , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Ender Konukoglu,et al.  PHiSeg: Capturing Uncertainty in Medical Image Segmentation , 2019, MICCAI.

[40]  Sotirios A. Tsaftaris,et al.  Medical Image Computing and Computer Assisted Intervention , 2017 .

[41]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[42]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[43]  T. Seierstad,et al.  Measurements of carotid intima media thickness in non-invasive high-frequency ultrasound images: the effect of dynamic range setting , 2015, Cardiovascular Ultrasound.

[44]  Ming-Hsuan Yang,et al.  SegFlow: Joint Learning for Video Object Segmentation and Optical Flow , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[45]  O. Ozturk,et al.  Relation Between Coronary Artery Disease, Risk Factors and Intima-Media Thickness of Carotid Artery, Arterial Distensibility, and Stiffness Index , 2003, Angiology.

[46]  Konstantina S. Nikita,et al.  Comparison of Block Matching and Differential Methods for Motion Analysis of the Carotid Artery Wall From Ultrasound Images , 2012, IEEE Transactions on Information Technology in Biomedicine.