MPWC-Net++: evolution of optical flow pyramidal convolutional neural network for ultrasound elastography

Recently, Convolutional Neural Networks (CNNs) have been very successful in optical flow estimation in computer vision. UltraSound Elastography (USE) displacement estimation step can be performed by optical flow CNNs. However, there is a large domain gap between ultrasound Radio-Frequency (RF) data and computer vision images which reduces the overall accuracy of displacement estimation. Some modifications of the network architecture are required to be able to extract reliable information from RF data. Modified Pyramidal Network (MPWC-Net) which is based on the well-known PWC-Net was among the first attempts that adopts the optical flow CNNs to USE displacement estimation. However, MPWC-Net suffers from several shortcomings that limit its application especially for unsupervised training. In this paper, we propose additional modifications to substantially improve MPWC-Net. We also publicly released the network’s trained weights.

[1]  Xiaoou Tang,et al.  LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  H Rivaz,et al.  Combining Total Variation Regularization with Window-Based Time Delay Estimation in Ultrasound Elastography , 2019, IEEE Transactions on Medical Imaging.

[3]  Hassan Rivaz,et al.  Semi-supervised Training of Optical Flow Convolutional Neural Networks in Ultrasound Elastography , 2020, MICCAI.

[4]  M. Chammas,et al.  Ultrasound Elastography: Review of Techniques and Clinical Applications , 2017, Theranostics.

[5]  T. Krouskop,et al.  Elastography: Ultrasonic estimation and imaging of the elastic properties of tissues , 1999, Proceedings of the Institution of Mechanical Engineers. Part H, Journal of engineering in medicine.

[6]  Hassan Rivaz,et al.  Real-time and High Quality Ultrasound Elastography Using Convolutional Neural Network by Incorporating Analytic Signal , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[7]  Gregory D. Hager,et al.  Real-Time Regularized Ultrasound Elastography , 2011, IEEE Transactions on Medical Imaging.

[8]  Stefan Roth,et al.  Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Hassan Rivaz,et al.  Displacement Estimation in Ultrasound Elastography Using Pyramidal Convolutional Neural Network , 2020, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[10]  An Unsupervised Approach to Ultrasound Elastography with End-to-end Strain Regularisation , 2020, MICCAI.

[11]  Russell H. Taylor,et al.  System for robot-assisted real-time laparoscopic ultrasound elastography , 2012, Medical Imaging.

[12]  Bo Peng,et al.  A Convolution Neural Network-Based Speckle Tracking Method for Ultrasound Elastography , 2018, 2018 IEEE International Ultrasonics Symposium (IUS).

[13]  Jan Kautz,et al.  PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Hassan Rivaz,et al.  GLUENet: Ultrasound Elastography Using Convolutional Neural Network , 2018, POCUS/BIVPCS/CuRIOUS/CPM@MICCAI.

[15]  Quan Zhang,et al.  Neural-network-based Motion Tracking for Breast Ultrasound Strain Elastography: An Initial Assessment of Performance and Feasibility , 2020, Ultrasonic imaging.

[16]  Lena Maier-Hein,et al.  Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation: International Workshops, POCUS 2018, BIVPCS 2018, CuRIOUS 2018, and CPM 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16–20, 2018, Proceedings , 2018, POCUS/BIVPCS/CuRIOUS/CPM@MICCAI.

[17]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[18]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).