Deep Learning in Ultrasound Elastography Imaging

It is known that changes in the mechanical properties of tissues are associated with the onset and progression of certain diseases. Ultrasound elastography is a technique to characterize tissue stiffness using ultrasound imaging either by measuring tissue strain using quasi-static elastography or natural organ pulsation elastography, or by tracing a propagated shear wave induced by a source or a natural vibration using dynamic elastography. In recent years, deep learning has begun to emerge in ultrasound elastography research. In this review, several common deep learning frameworks in the computer vision community, such as multilayer perceptron, convolutional neural network, and recurrent neural network are described. Then, recent advances in ultrasound elastography using such deep learning techniques are revisited in terms of algorithm development and clinical diagnosis. Finally, the current challenges and future developments of deep learning in ultrasound elastography are prospected.

[1]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[2]  Nima Tajbakhsh,et al.  Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? , 2016, IEEE Transactions on Medical Imaging.

[3]  Samuel Hybois,et al.  Ultrasound Shear Wave Viscoelastography: Model-Independent Quantification of the Complex Shear Modulus , 2016, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[4]  M Fink,et al.  A solution to diffraction biases in sonoelasticity: the acoustic impulse technique. , 1999, The Journal of the Acoustical Society of America.

[5]  K. Bowman Mechanical Behavior of Materials , 2003 .

[6]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Woo Kyung Moon,et al.  Comparison of Ultrasound Elastography and Color Doppler Ultrasonography for Distinguishing Small Triple‐Negative Breast Cancer From Fibroadenoma , 2018, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[8]  R. Barr,et al.  Future of breast elastography , 2019, Ultrasonography.

[9]  J. Ophir,et al.  Elastography: A Quantitative Method for Imaging the Elasticity of Biological Tissues , 1991, Ultrasonic imaging.

[10]  M. Fink,et al.  Supersonic shear imaging: a new technique for soft tissue elasticity mapping , 2004, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[11]  Ge Wang,et al.  A Perspective on Deep Imaging , 2016, IEEE Access.

[12]  Ilias Gatos,et al.  Temporal stability assessment in shear wave elasticity images validated by deep learning neural network for chronic liver disease fibrosis stage assessment. , 2019, Medical physics.

[13]  J. Greenleaf,et al.  Remote measurement of material properties from radiation force induced vibration of an embedded sphere. , 2002, The Journal of the Acoustical Society of America.

[14]  Bo Peng,et al.  A Convolution Neural Network-Based Speckle Tracking Method for Ultrasound Elastography , 2018, 2018 IEEE International Ultrasonics Symposium (IUS).

[15]  Kang Ryoung Park,et al.  Artificial Intelligence-Based Thyroid Nodule Classification Using Information from Spatial and Frequency Domains , 2019, Journal of clinical medicine.

[16]  Anthony E. Samir,et al.  Shear wave elastography in chronic kidney disease: a pilot experience in native kidneys , 2015, BMC Nephrology.

[17]  Michael J. Bey,et al.  Clinical utilization of shear wave elastography in the musculoskeletal system , 2018, Ultrasonography.

[18]  Christophe K. Mannaerts,et al.  Automated multiparametric localization of prostate cancer based on B-mode, shear-wave elastography, and contrast-enhanced ultrasound radiomics , 2019, European Radiology.

[19]  S. Salcudean,et al.  Viscoelastic parameter estimation based on spectral analysis , 2008, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[20]  Lipo Wang,et al.  Deep Learning Applications in Medical Image Analysis , 2018, IEEE Access.

[21]  Manojit Pramanik,et al.  Deep neural network-based bandwidth enhancement of photoacoustic data , 2017, Journal of biomedical optics.

[22]  Stanislav Emelianov,et al.  Biophysical Bases of Elasticity Imaging , 1995 .

[23]  Raymond T Chung,et al.  Shear-wave elastography for the estimation of liver fibrosis in chronic liver disease: determining accuracy and ideal site for measurement. , 2015, Radiology.

[24]  Albert Montillo,et al.  Deep learning convolutional neural networks for the estimation of liver fibrosis severity from ultrasound texture , 2019, Medical Imaging.

[25]  Lin Yang,et al.  Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation , 2016, NIPS.

[26]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[27]  Qi Zhang,et al.  Deep learning based classification of breast tumors with shear-wave elastography. , 2016, Ultrasonics.

[28]  Patrick Segers,et al.  An in silico framework to analyze the anisotropic shear wave mechanics in cardiac shear wave elastography , 2018, Physics in medicine and biology.

[29]  Yuu Ono,et al.  A method to reduce the influence of reflected waves on shear velocity measurements using B-mode scanning time delay , 2015 .

[30]  J. Ophir,et al.  Myocardial elastography--a feasibility study in vivo. , 2002, Ultrasound in medicine & biology.

[31]  Damien Garcia,et al.  Noninvasive Vascular Elastography With Plane Strain Incompressibility Assumption Using Ultrafast Coherent Compound Plane Wave Imaging , 2015, IEEE Transactions on Medical Imaging.

[32]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[33]  Stuart Paterson,et al.  Endoscopic ultrasound-guided elastography in the nodal staging of oesophageal cancer. , 2012, World journal of gastroenterology.

[34]  Purang Abolmaesumi,et al.  Deep Recurrent Neural Networks for Prostate Cancer Detection: Analysis of Temporal Enhanced Ultrasound , 2018, IEEE Transactions on Medical Imaging.

[35]  Sanghamithra Korukonda,et al.  Visualizing the radial and circumferential strain distribution within vessel phantoms using synthetic-aperture ultrasound elastography , 2012, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[36]  T. Krouskop,et al.  Elastography: Ultrasonic estimation and imaging of the elastic properties of tissues , 1999, Proceedings of the Institution of Mechanical Engineers. Part H, Journal of engineering in medicine.

[37]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[38]  Hairong Zheng,et al.  Computer-aided diagnosis based on quantitative elastographic features with supersonic shear wave imaging. , 2014, Ultrasound in medicine & biology.

[39]  Runmin Wei,et al.  Clinical prediction of HBV and HCV related hepatic fibrosis using machine learning , 2018, EBioMedicine.

[40]  E E Konofagou,et al.  Stochastic precision analysis of 2D cardiac strain estimation in vivo , 2014, Physics in medicine and biology.

[41]  Hiroshi Kanai,et al.  Elasticity Imaging of Atheroma With Transcutaneous Ultrasound , 2003, Circulation.

[42]  Denis Friboulet,et al.  High-Quality Plane Wave Compounding Using Convolutional Neural Networks , 2017, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[43]  Y. Yamakoshi,et al.  Ultrasonic imaging of internal vibration of soft tissue under forced vibration , 1990, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[44]  Cheng Li,et al.  A Radiomics Approach With CNN for Shear-Wave Elastography Breast Tumor Classification , 2018, IEEE Transactions on Biomedical Engineering.

[45]  Richard G Barr Liver Elastography Still in Its Infancy. , 2018, Radiology.

[46]  Christoph F Dietrich,et al.  Endoscopic ultrasound-guided fine-needle aspiration biopsy and trucut biopsy in gastroenterology - An overview. , 2009, Best practice & research. Clinical gastroenterology.

[47]  Jie Tian,et al.  Deep learning Radiomics of shear wave elastography significantly improved diagnostic performance for assessing liver fibrosis in chronic hepatitis B: a prospective multicentre study , 2018, Gut.

[48]  Jianwen Luo,et al.  Learning the implicit strain reconstruction in ultrasound elastography using privileged information , 2019, Medical Image Anal..

[49]  Xiaoou Tang,et al.  LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[50]  Xiang Liu,et al.  Learning to Diagnose Cirrhosis with Liver Capsule Guided Ultrasound Image Classification , 2017, Sensors.

[51]  J. Greenleaf,et al.  Ultrasound-stimulated vibro-acoustic spectrography. , 1998, Science.

[52]  Shigao Chen,et al.  Shearwave dispersion ultrasound vibrometry (SDUV) for measuring tissue elasticity and viscosity , 2009, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[53]  K J Parker,et al.  Imaging of the elastic properties of tissue--a review. , 1996, Ultrasound in medicine & biology.

[54]  Guy Cloutier,et al.  Two‐dimensional affine model‐based estimators for principal strain vascular ultrasound elastography with compound plane wave and transverse oscillation beamforming , 2019, Ultrasonics.

[55]  Michael J. Black,et al.  Optical Flow Estimation Using a Spatial Pyramid Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  James F. Greenleaf,et al.  Application of Acoustoelasticity to Evaluate Nonlinear Modulus in Ex Vivo Kidneys , 2018, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[57]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[58]  Qi Zhang,et al.  Dual-modal computer-assisted evaluation of axillary lymph node metastasis in breast cancer patients on both real-time elastography and B-mode ultrasound. , 2017, European journal of radiology.

[59]  J. Jensen,et al.  A new method for estimation of velocity vectors , 1998, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[60]  P. Vilmann,et al.  Efficacy of an artificial neural network-based approach to endoscopic ultrasound elastography in diagnosis of focal pancreatic masses. , 2012, Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association.

[61]  C. D'Orsi,et al.  Influence of computer-aided detection on performance of screening mammography. , 2007, The New England journal of medicine.

[62]  Panagiotis Papadimitroulas,et al.  Deep learning networks on chronic liver disease assessment with fine-tuning of shear wave elastography image sequences , 2020, Physics in medicine and biology.

[63]  Guy Cloutier,et al.  Ultrasound Elastography and MR Elastography for Assessing Liver Fibrosis: Part 2, Diagnostic Performance, Confounders, and Future Directions. , 2015, AJR. American journal of roentgenology.

[64]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[65]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[66]  Erik Smistad,et al.  Automatic Myocardial Strain Imaging in Echocardiography Using Deep Learning , 2018, DLMIA/ML-CDS@MICCAI.

[67]  Hariharan Ravishankar,et al.  Understanding the Mechanisms of Deep Transfer Learning for Medical Images , 2016, LABELS/DLMIA@MICCAI.

[68]  Arinc Ozturk,et al.  Liver fibrosis imaging: A clinical review of ultrasound and magnetic resonance elastography , 2020, Journal of magnetic resonance imaging : JMRI.

[69]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[70]  Jianwen Luo,et al.  Direct Reconstruction of Ultrasound Elastography Using an End-to-End Deep Neural Network , 2018, MICCAI.

[71]  R. Memo,et al.  Shear Wave Ultrasound Elastography of the Prostate: Initial Results , 2012, Ultrasound quarterly.

[72]  Anil T Ahuja,et al.  Shear wave elasticity imaging of cervical lymph nodes. , 2012, Ultrasound in medicine & biology.

[73]  N Bom,et al.  Characterization of plaque components and vulnerability with intravascular ultrasound elastography. , 2000, Physics in medicine and biology.

[74]  Wenming Cao,et al.  Liver Fibrosis Classification Based on Transfer Learning and FCNet for Ultrasound Images , 2017, IEEE Access.

[75]  C. Le Berre,et al.  Application of Artificial Intelligence to Gastroenterology and Hepatology. , 2019, Gastroenterology.

[76]  F. Sebag,et al.  Shear wave elastography: a new ultrasound imaging mode for the differential diagnosis of benign and malignant thyroid nodules. , 2010, The Journal of clinical endocrinology and metabolism.

[77]  F. Sardanelli,et al.  Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine , 2018, European Radiology Experimental.

[78]  A. Goddi,et al.  Breast elastography: A literature review. , 2012, Journal of ultrasound.

[79]  H. Oestreicher Field and Impedance of an Oscillating Sphere in a Viscoelastic Medium with an Application to Biophysics , 1951 .

[80]  G. Trahey,et al.  On the feasibility of remote palpation using acoustic radiation force. , 2001, The Journal of the Acoustical Society of America.

[81]  Qi Wei,et al.  Artificial intelligence in medical imaging of the liver , 2019, World journal of gastroenterology.

[82]  Guillermo Rus-Carlborg,et al.  Why Are Viscosity and Nonlinearity Bound to Make an Impact in Clinical Elastographic Diagnosis? , 2020, Sensors.

[83]  Tsuyoshi Shiina,et al.  WFUMB guidelines and recommendations for clinical use of ultrasound elastography: Part 1: basic principles and terminology. , 2015, Ultrasound in medicine & biology.

[84]  M. Tanter,et al.  Investigating liver stiffness and viscosity for fibrosis, steatosis and activity staging using shear wave elastography. , 2015, Journal of hepatology.

[85]  M F Insana,et al.  Viscoelasticity imaging using ultrasound: parameters and error analysis. , 2007, Physics in medicine and biology.

[86]  Lei Xu,et al.  A Deep Siamese-Based Plantar Fasciitis Classification Method Using Shear Wave Elastography , 2019, IEEE Access.

[87]  S. Emelianov,et al.  Shear wave elasticity imaging: a new ultrasonic technology of medical diagnostics. , 1998, Ultrasound in medicine & biology.

[88]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[89]  Dong Ni,et al.  Deep Learning in Medical Ultrasound Analysis: A Review , 2019, Engineering.

[90]  Hui Wang,et al.  Value of ultrasound elastography in assessment of enlarged cervical lymph nodes. , 2012, Asian Pacific journal of cancer prevention : APJCP.

[91]  M Tanter,et al.  In vivo evaluation of the elastic anisotropy of the human Achilles tendon using shear wave dispersion analysis , 2014, Physics in medicine and biology.

[92]  Jan Kautz,et al.  PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[93]  G. Soulez,et al.  Investigation of out-of-plane motion artifacts in 2D noninvasive vascular ultrasound elastography , 2018, Physics in medicine and biology.

[94]  K. Parker,et al.  "Sonoelasticity" images derived from ultrasound signals in mechanically vibrated tissues. , 1990, Ultrasound in medicine & biology.

[95]  Jianchao Zeng,et al.  Diagnosis of Benign and Malignant Thyroid Nodules Using Combined Conventional Ultrasound and Ultrasound Elasticity Imaging , 2020, IEEE Journal of Biomedical and Health Informatics.

[96]  Jasjit S. Suri,et al.  Symtosis: A liver ultrasound tissue characterization and risk stratification in optimized deep learning paradigm , 2018, Comput. Methods Programs Biomed..

[97]  M.E. Aderson,et al.  Multi-dimensional velocity estimation with ultrasound using spatial quadrature , 1998, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[98]  S. Salcudean,et al.  Identifying the mechanical properties of tissue by ultrasound strain imaging. , 2006, Ultrasound in medicine & biology.

[99]  Guy Cloutier,et al.  Parameterized Strain Estimation for Vascular Ultrasound Elastography With Sparse Representation , 2020, IEEE Transactions on Medical Imaging.

[100]  Robert F. Ling,et al.  A computer generated aid for cluster analysis , 1973, CACM.

[101]  Richard G P Lopata,et al.  Three-dimensional cardiac strain imaging in healthy children using RF-data. , 2011, Ultrasound in medicine & biology.

[102]  Heng Zhao,et al.  Two-dimensional shear-wave elastography on conventional ultrasound scanners with time-aligned sequential tracking (TAST) and comb-push ultrasound shear elastography (CUSE) , 2015, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[103]  D. Rubens,et al.  Imaging the elastic properties of tissue: the 20 year perspective , 2011, Physics in medicine and biology.

[104]  N Bom,et al.  Morphological and mechanical information of coronary arteries obtained with intravascular elastography; feasibility study in vivo. , 2002, European heart journal.

[105]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[106]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[107]  Muyinatu A. Lediju Bell,et al.  Deep Learning to Obtain Simultaneous Image and Segmentation Outputs From a Single Input of Raw Ultrasound Channel Data , 2020, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[108]  Hassan Rivaz,et al.  Displacement Estimation in Ultrasound Elastography Using Pyramidal Convolutional Neural Network , 2020, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[109]  James S. Duncan,et al.  Learning-Based Spatiotemporal Regularization and Integration of Tracking Methods for Regional 4D Cardiac Deformation Analysis , 2017, MICCAI.

[110]  A. Thompson,et al.  Invasive breast cancer: relationship between shear-wave elastographic findings and histologic prognostic factors. , 2012, Radiology.

[111]  Ilias Gatos,et al.  A Machine-Learning Algorithm Toward Color Analysis for Chronic Liver Disease Classification, Employing Ultrasound Shear Wave Elastography. , 2017, Ultrasound in medicine & biology.

[112]  Phaneendra K. Yalavarthy,et al.  PA-Fuse: deep supervised approach for the fusion of photoacoustic images with distinct reconstruction characteristics. , 2019, Biomedical optics express.

[113]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[114]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[115]  Makoto Yamakawa,et al.  Current status and perspectives for computer-aided ultrasonic diagnosis of liver lesions using deep learning technology , 2019, Hepatology International.

[116]  Hassan Rivaz,et al.  GLUENet: Ultrasound Elastography Using Convolutional Neural Network , 2018, POCUS/BIVPCS/CuRIOUS/CPM@MICCAI.

[117]  S. Patel,et al.  Sequence to Sequence Learning in Neural Network , 2017 .

[118]  J. Greenleaf,et al.  Selected methods for imaging elastic properties of biological tissues. , 2003, Annual review of biomedical engineering.

[119]  Herbert Reismann,et al.  Elasticity: Theory and Applications , 1980 .

[120]  Assad A. Oberai,et al.  Circumventing the solution of inverse problems in mechanics through deep learning: Application to elasticity imaging , 2019, Computer Methods in Applied Mechanics and Engineering.

[121]  Shuang Song,et al.  Dual-mode artificially-intelligent diagnosis of breast tumours in shear-wave elastography and B-mode ultrasound using deep polynomial networks. , 2019, Medical engineering & physics.

[122]  Jarrod Orszulak,et al.  Shear-Modulus Estimation by Application of Spatially-Modulated Impulsive Acoustic Radiation Force , 2007, Ultrasonic imaging.

[123]  Bradley J Erickson,et al.  A Survey of Deep-Learning Applications in Ultrasound: Artificial Intelligence-Powered Ultrasound for Improving Clinical Workflow. , 2019, Journal of the American College of Radiology : JACR.

[124]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[125]  Guy Cloutier,et al.  Reconstruction of Viscosity Maps in Ultrasound Shear Wave Elastography , 2019, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[126]  Yoshua Bengio,et al.  Maxout Networks , 2013, ICML.

[127]  Richard G. P. Lopata,et al.  Noninvasive Carotid Strain Imaging Using Angular Compounding at Large Beam Steered Angles: Validation in Vessel Phantoms , 2009, IEEE Transactions on Medical Imaging.

[128]  K. Parker,et al.  Sono-Elasticity: Medical Elasticity Images Derived from Ultrasound Signals in Mechanically Vibrated Targets , 1988 .

[129]  Quan Zhang,et al.  Neural-network-based Motion Tracking for Breast Ultrasound Strain Elastography: An Initial Assessment of Performance and Feasibility , 2020, Ultrasonic imaging.

[130]  Michel Bertrand,et al.  Noninvasive vascular elastography: theoretical framework , 2004, IEEE Transactions on Medical Imaging.