Plane-Wave Image Reconstruction via Generative Adversarial Network and Attention Mechanism

Plane-wave imaging (PWI) has been designed and widely applied in ultrasound imaging in recent years. This technique achieves high frame rate while compromising with the image quality, such as spatial resolution and contrast. Although the compounding method is capable to improve reconstruction effect, this method limits the temporal resolution. The existing generative adversarial network (GAN)-based methods are not capable to fully extract shallow, high-frequency, and high-level information. To reconstruct high-resolution plane-wave images, we propose a reconstruction method via attention mechanism and Unet-based GAN (AUGAN). The generator of AUGAN adopts the Unet-based architecture that containing skip concatenate pathways, which can capture low-level features of different scales. Besides, in order to excavate image information comprehensively, we adopt the pixel attention to strengthen the extraction of shallow features and local aware attention to put more attention to limited high-frequency features. Moreover, perceptual loss is introduced to make the model keep more high-level information of target images. The proposed method are evaluated on datasets acquired from simulation, experimental phantom, and in vivo targets provided by the PWI Challenge in Medical Ultrasound (PICMUS). AUGAN can achieve maximal improvements of full-width at half-maximum (FWHM) by 8.27% (simulation) and 33.21% (experiment) compared with coherent plane-wave compounding (CPWC) while 2.46% (simulation) and 5.05% (experiment) compared with Unet-based GAN (UnetGAN). The maximum contrast ratio (CR) improvements are 11.32% upon CPWC and 2.3% upon UnetGAN. For in vivo data, AUGAN shows its advantage in preserving anatomical structures compared with other state-of-the-art methods in plane-wave image reconstruction. The attention ablation experiment illustrates that both pixel attention and local aware attention have further improved the plane-wave image reconstruction.

[1]  Yongdong Zhang,et al.  Potential of Attention Mechanism for Classification of Optical Coherence Tomography Images , 2018, 2018 IEEE Visual Communications and Image Processing (VCIP).

[2]  Mingyue Ding,et al.  Image Quality Assessment Based on Regional Mutual Information , 2011, 2011 International Conference on Intelligent Computation and Bio-Medical Instrumentation.

[3]  Zhaorong Zong,et al.  On Application of Natural Language Processing in Machine Translation , 2018, 2018 3rd International Conference on Mechanical, Control and Computer Engineering (ICMCCE).

[4]  M. Fink,et al.  Coherent plane-wave compounding for very high frame rate ultrasonography and transient elastography , 2009, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[5]  Juan Cheng,et al.  EEG-Based Emotion Recognition via Channel-Wise Attention and Self Attention , 2023, IEEE Transactions on Affective Computing.

[6]  A. Mahloojifar,et al.  Minimum variance beamforming combined with adaptive coherence weighting applied to medical ultrasound imaging , 2009, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[7]  H. Torp,et al.  The Generalized Contrast-to-Noise Ratio: A Formal Definition for Lesion Detectability , 2019, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[8]  Alexander V. Terekhov,et al.  Gradual Tuning: a better way of Fine Tuning the parameters of a Deep Neural Network , 2017, ArXiv.

[9]  Alexander J. Smola,et al.  Stacked Attention Networks for Image Question Answering , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Jianwen Luo,et al.  High-Quality Reconstruction of Plane-Wave Imaging Using Generative Adversarial Network , 2018, 2018 IEEE International Ultrasonics Symposium (IUS).

[11]  Pai-Chi Li,et al.  Adaptive imaging using the generalized coherence factor , 2003, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[12]  Mickael Tanter,et al.  Ultrafast imaging in biomedical ultrasound , 2014, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[13]  Tania Stathaki,et al.  Salient Object Detection Combining a Self-attention Module and a Feature Pyramid Network , 2020, Electronics.

[14]  J. Capon.  High-resolution frequency-wavenumber spectrum analysis , 1969 .

[15]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[16]  Matiur Rahman Minar,et al.  Recent Advances in Deep Learning: An Overview , 2018, ArXiv.

[17]  Jong Chul Ye,et al.  Efficient B-Mode Ultrasound Image Reconstruction From Sub-Sampled RF Data Using Deep Learning , 2017, IEEE Transactions on Medical Imaging.

[18]  Taesung Park,et al.  CyCADA: Cycle-Consistent Adversarial Domain Adaptation , 2017, ICML.

[19]  Song Guo,et al.  MAANet: Multi-view Aware Attention Networks for Image Super-Resolution , 2019, ArXiv.

[20]  Steven C. H. Hoi,et al.  Deep Learning for Image Super-Resolution: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Anne E. C. M. Saris,et al.  Cardiac Motion Estimation Using Ultrafast Ultrasound Imaging Tested in a Finite Element Model of Cardiac Mechanics , 2015, FIMH.

[22]  M. Fink,et al.  Ultrasound contrast plane wave imaging , 2012, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[23]  M. O'Donnell,et al.  Coherence factor of speckle from a multi-row probe , 1999, 1999 IEEE Ultrasonics Symposium. Proceedings. International Symposium (Cat. No.99CH37027).

[24]  Austin Reiter,et al.  A Generative Adversarial Neural Network for Beamforming Ultrasound Images Invited Presentation , 2019, 2019 53rd Annual Conference on Information Sciences and Systems (CISS).

[25]  Paolo Torroni,et al.  Attention in Natural Language Processing , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Dongwoon Hyun,et al.  Challenge on Ultrasound Beamforming with Deep Learning (CUBDL) , 2020, 2020 IEEE International Ultrasonics Symposium (IUS).

[27]  Yu Qiao,et al.  Efficient Image Super-Resolution Using Pixel Attention , 2020, ECCV Workshops.

[28]  Yuanyuan Wang,et al.  Image Quality Enhancement Using a Deep Neural Network for Plane Wave Medical Ultrasound Imaging , 2020, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[29]  Yonina C. Eldar,et al.  Deep Learning in Ultrasound Imaging , 2019, Proceedings of the IEEE.

[30]  Hongtao Huo,et al.  AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks , 2021, IEEE Transactions on Multimedia.

[31]  J. Arendt.  Paper presented at the 10th Nordic-Baltic Conference on Biomedical Imaging: Field: A Program for Simulating Ultrasound Systems , 1996 .

[32]  Zhen Wang,et al.  On the Effectiveness of Least Squares Generative Adversarial Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Mohammed Ghanbari,et al.  Scope of validity of PSNR in image/video quality assessment , 2008 .

[34]  Jinhua Yu,et al.  High Spatial–Temporal Resolution Reconstruction of Plane-Wave Ultrasound Images With a Multichannel Multiscale Convolutional Neural Network , 2018, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[35]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[36]  Nikolaos Doulamis,et al.  Deep Learning for Computer Vision: A Brief Review , 2018, Comput. Intell. Neurosci..

[37]  Lei Xie,et al.  Attention-Based End-to-End Speech Recognition on Voice Search , 2017, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  J. Jensen,et al.  Calculation of pressure fields from arbitrarily shaped, apodized, and excited ultrasound transducers , 1992, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control.

[39]  Taghi M. Khoshgoftaar,et al.  A survey on Image Data Augmentation for Deep Learning , 2019, Journal of Big Data.

[40]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[41]  Matthew Berger,et al.  Domain Adaptation for Ultrasound Beamforming , 2020, MICCAI.

[42]  Jianwen Luo,et al.  Ultrasound image reconstruction from plane wave radio-frequency data by self-supervised deep neural network , 2021, Medical Image Anal..

[43]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[44]  Denis Friboulet,et al.  High-Quality Plane Wave Compounding Using Convolutional Neural Networks , 2017, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[45]  C. Blehm,et al.  Computer vision syndrome: a review. , 2005, Survey of ophthalmology.

[46]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[47]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[48]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[49]  O. Bernard,et al.  Plane-Wave Imaging Challenge in Medical Ultrasound , 2016, 2016 IEEE International Ultrasonics Symposium (IUS).