Deep learning for photoacoustic imaging: a survey

Machine learning has been developed dramatically and witnessed a lot of applications in various fields over the past few years. This boom originated in 2009, when a new model emerged, that is, the deep artificial neural network, which began to surpass other established mature models on some important benchmarks. Later, it was widely used in academia and industry. Ranging from image analysis to natural language processing, it fully exerted its magic and now become the state-of-the-art machine learning models. Deep neural networks have great potential in medical imaging technology, medical data analysis, medical diagnosis and other healthcare issues, and is promoted in both pre-clinical and even clinical stages. In this review, we performed an overview of some new developments and challenges in the application of machine learning to medical image analysis, with a special focus on deep learning in photoacoustic imaging. The aim of this review is threefold: (i) introducing deep learning with some important basics, (ii) reviewing recent works that apply deep learning in the entire ecological chain of photoacoustic imaging, from image reconstruction to disease diagnosis, (iii) providing some open source materials and other resources for researchers interested in applying deep learning to photoacoustic imaging.

[1]  Geoffrey P. Luke,et al.  O-Net: A Convolutional Neural Network for Quantitative Photoacoustic Image Segmentation and Oximetry , 2019, 1911.01935.

[2]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[3]  Tong Lu,et al.  A deep learning method based on U-Net for quantitative photoacoustic imaging , 2020, BiOS.

[4]  Rayyan Manwar,et al.  Deep learning protocol for improved photoacoustic brain imaging , 2020, Journal of biophotonics.

[5]  Lihong V. Wang,et al.  Photoacoustic Tomography: In Vivo Imaging from Organelles to Organs , 2012, Science.

[6]  Israr Ul Haq,et al.  Convolutional autoencoder-based reconstruction of vascular structures in photoacoustic images , 2020 .

[7]  Chulhong Kim,et al.  Deep learning-based speed of sound aberration correction in photoacoustic images , 2020, BiOS.

[8]  Junjie Yao,et al.  Feature article: A generative adversarial network for artifact removal in photoacoustic computed tomography with a linear-array transducer , 2020, Experimental biology and medicine.

[9]  Jesse V. Jokerst,et al.  Deep Learning Improves Contrast in Low-Fluence Photoacoustic Imaging , 2020, Biomedical optics express.

[10]  Jianwen Luo,et al.  End-to-end deep neural network for optical inversion in quantitative photoacoustic imaging. , 2018, Optics letters.

[11]  Julianna M Czum,et al.  Dive Into Deep Learning. , 2020, Journal of the American College of Radiology : JACR.

[12]  Arvid Lundervold,et al.  An overview of deep learning in medical imaging focusing on MRI , 2018, Zeitschrift fur medizinische Physik.

[13]  Patrick Putzky,et al.  Recurrent inference machines for accelerated MRI reconstruction. , 2018 .

[14]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[15]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.

[16]  Fei Gao,et al.  EDA-Net: Dense Aggregation of Deep and Shallow Information Achieves Quantitative Photoacoustic Blood Oxygenation Imaging Deep in Human Breast , 2019, MICCAI.

[17]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[18]  Stephan Antholzer,et al.  Deep learning for photoacoustic tomography from sparse data , 2017, Inverse problems in science and engineering.

[19]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[20]  Navalgund Rao,et al.  Deep 3D convolutional neural network for automatic cancer tissue detection using multispectral photoacoustic imaging , 2019, Medical Imaging.

[21]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Aggelos K. Katsaggelos,et al.  Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods , 2018, IEEE Signal Processing Magazine.

[24]  Lihong V. Wang,et al.  Tutorial on Photoacoustic Microscopy and Computed Tomography , 2008, IEEE Journal of Selected Topics in Quantum Electronics.

[25]  Asifullah Khan,et al.  A survey of the recent architectures of deep convolutional neural networks , 2019, Artificial Intelligence Review.

[26]  Vasilis Ntziachristos,et al.  Deep Learning-Based Spectral Unmixing for Optoacoustic Imaging of Tissue Oxygen Saturation , 2020, IEEE Transactions on Medical Imaging.

[27]  Fei Gao,et al.  Quantitative Photoacoustic Blood Oxygenation Imaging Using Deep Residual And Recurrent Neural Network , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[28]  Steven Guan,et al.  Limited-View and Sparse Photoacoustic Tomography for Neuroimaging with Deep Learning , 2020, Scientific Reports.

[29]  Meng Zhou,et al.  Photoacoustic Image Classification and Segmentation of Breast Cancer: A Feasibility Study , 2019, IEEE Access.

[30]  Jin U. Kang,et al.  Enabling fast and high quality LED photoacoustic imaging: a recurrent neural networks based approach. , 2018, Biomedical optics express.

[31]  P. Beard Biomedical photoacoustic imaging , 2011, Interface Focus.

[32]  Phaneendra K. Yalavarthy,et al.  PA-Fuse: deep supervised approach for the fusion of photoacoustic images with distinct reconstruction characteristics. , 2019, Biomedical optics express.

[33]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[34]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[35]  Janek Gröhl,et al.  Reconstruction of initial pressure from limited view photoacoustic images using deep learning , 2018, BiOS.

[36]  Sri-Rajasekhar Kothapalli,et al.  Unsupervised deep learning approach for photoacoustic spectral unmixing , 2020, BiOS.

[37]  Bastien Arnal,et al.  Solving the visibility problem in photoacoustic imaging with a deep learning approach providing prediction uncertainties , 2020, 2006.13096.

[38]  Luo Jianwen,et al.  Machine-learning enhanced photoacoustic computed tomography in a limited view configuration , 2019 .

[39]  Trevor Darrell,et al.  Deep Layer Aggregation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Lihong V. Wang,et al.  In vivo functional photoacoustic microscopy of cutaneous microvasculature in human skin. , 2011, Journal of biomedical optics.

[41]  Susumu Saito,et al.  Semi-Supervised Learning With Structured Knowledge For Body Hair Detection In Photoacoustic Image , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[42]  Lediju Bell,et al.  Deep learning the sound of light to guide surgeries , 2019 .

[43]  Stephan Antholzer,et al.  Learned backprojection for sparse and limited view photoacoustic tomography , 2019, BiOS.

[44]  Stephan Antholzer,et al.  Deep Learning Versus $\ell^{1}$ -Minimization for Compressed Sensing Photoacoustic Tomography , 2018, 2018 IEEE International Ultrasonics Symposium (IUS).

[45]  Shenghua Gao,et al.  Ki-GAN: Knowledge Infusion Generative Adversarial Network for Photoacoustic Image Reconstruction In Vivo , 2019, MICCAI.

[46]  Muhammad Imran Razzak,et al.  Deep Learning for Medical Image Processing: Overview, Challenges and Future , 2017, ArXiv.

[47]  Quing Zhu,et al.  A real-time photoacoustic tomography system for small animals. , 2009, Optics express.

[48]  Noboru Murata,et al.  Neural Network with Unbounded Activation Functions is Universal Approximator , 2015, 1505.03654.

[49]  Raymond W. Ptucha,et al.  Prostate cancer detection using photoacoustic imaging and deep learning , 2016, Image Processing: Algorithms and Systems.

[50]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[51]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Bhargava Chinni,et al.  Computer aided detection of prostate cancer using multiwavelength photoacoustic data with convolutional neural network , 2020, Biomed. Signal Process. Control..

[53]  Jiasheng Zhou,et al.  Photoacoustic Microscopy with Sparse Data Enabled by Convolutional Neural Networks for Fast Imaging , 2020, ArXiv.

[54]  Chloé Audigier,et al.  Robust Photoacoustic Beamforming Using Dense Convolutional Neural Networks , 2018, POCUS/BIVPCS/CuRIOUS/CPM@MICCAI.

[55]  Stephan Antholzer,et al.  Real-time photoacoustic projection imaging using deep learning , 2018, 1801.06693.

[56]  Muyinatu A. Lediju Bell,et al.  Deep learning to detect catheter tips in vivo during photoacoustic-guided catheter interventions : Invited Presentation , 2019, 2019 53rd Annual Conference on Information Sciences and Systems (CISS).

[57]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[58]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[59]  Geoffrey P. Luke,et al.  absO2luteU-Net: Tissue Oxygenation Calculation Using Photoacoustic Imaging and Convolutional Neural Networks , 2019 .

[60]  Feng Gao,et al.  Human Breast Numerical Model Generation Based on Deep Learning for Photoacoustic Imaging , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[61]  Yoeri E Boink,et al.  A Partially-Learned Algorithm for Joint Photo-acoustic Reconstruction and Segmentation , 2019, IEEE Transactions on Medical Imaging.

[62]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[63]  Lei Xi,et al.  Deep-learning-based motion-correction algorithm in optical resolution photoacoustic microscopy , 2019, Visual Computing for Industry, Biomedicine, and Art.

[64]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[65]  Jin U. Kang,et al.  Towards a Fast and Safe LED-Based Photoacoustic Imaging Using Deep Convolutional Neural Network , 2018, MICCAI.

[66]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[67]  Bastien Arnal,et al.  Compensating for visibility artefacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties , 2020, Photoacoustics.

[68]  Jeffrey A. Fessler,et al.  Image Reconstruction is a New Frontier of Machine Learning , 2018, IEEE Transactions on Medical Imaging.

[69]  Paul C. Beard,et al.  Approximate k-space models and Deep Learning for fast photoacoustic reconstruction , 2018, MLMIR@MICCAI.

[70]  Max Welling,et al.  Data-driven Reconstruction of Gravitationally Lensed Galaxies Using Recurrent Inference Machines , 2019, The Astrophysical Journal.

[71]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[72]  Hui Li,et al.  Computer-aided classification system for early endometrial cancer of co-registered photoacoustic and ultrasonic signals , 2019 .

[73]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[74]  Muyinatu A. Lediju Bell,et al.  Photoacoustic Source Detection and Reflection Artifact Removal Enabled by Deep Learning , 2018, IEEE Transactions on Medical Imaging.

[75]  Xueding Wang,et al.  Adipocyte Size Evaluation Based on Photoacoustic Spectral Analysis Combined with Deep Learning Method , 2018, Applied Sciences.

[76]  Stephan Antholzer,et al.  Deep Learning of truncated singular values for limited view photoacoustic tomography , 2019, BiOS.

[77]  Navalgund Rao,et al.  Transfer learning for automatic cancer tissue detection using multispectral photoacoustic imaging , 2019, Medical Imaging.

[78]  Lihong V. Wang,et al.  A practical guide to photoacoustic tomography in the life sciences , 2016, Nature Methods.

[79]  Vasilis Ntziachristos,et al.  A sparse deep learning approach for automatic segmentation of human vasculature in multispectral optoacoustic tomography , 2020, Photoacoustics.

[80]  Sri-Rajasekhar Kothapalli,et al.  Novel deep learning architecture for optical fluence dependent photoacoustic target localization , 2019, BiOS.

[81]  Weimin Zhou,et al.  Generation of anatomically realistic numerical phantoms for photoacoustic and ultrasonic breast imaging , 2017, Journal of biomedical optics.

[82]  Fei Gao,et al.  Accelerated Photoacoustic Tomography Reconstruction via Recurrent Inference Machines , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[83]  Andreas Stolcke,et al.  The Microsoft 2017 Conversational Speech Recognition System , 2017, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[84]  Junjie Yao,et al.  Reconstructing Undersampled Photoacoustic Microscopy Images Using Deep Learning , 2020, IEEE Transactions on Medical Imaging.

[85]  S. Jacques Optical properties of biological tissues: a review , 2013, Physics in medicine and biology.

[86]  Vasilis Ntziachristos,et al.  Eigenspectra optoacoustic tomography achieves quantitative blood oxygenation imaging deep in tissues , 2015, Nature Communications.

[87]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Jinchao Feng,et al.  End-to-end Res-Unet based reconstruction algorithm for photoacoustic imaging. , 2020, Biomedical optics express.

[89]  Daniel George,et al.  Denoising Gravitational Waves with Enhanced Deep Recurrent Denoising Auto-encoders , 2017, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[90]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[91]  Heung-Il Suk,et al.  Deep Learning in Medical Image Analysis. , 2017, Annual review of biomedical engineering.

[92]  Zheng Sun,et al.  A Deep Learning Method for Limited-View Intravascular Photoacoustic Image Reconstruction , 2020 .

[93]  Matthew O'Donnell,et al.  Deep-Learning Image Reconstruction for Real-Time Photoacoustic System , 2020, IEEE Transactions on Medical Imaging.

[94]  Kenji Suzuki,et al.  Overview of deep learning in medical imaging , 2017, Radiological Physics and Technology.

[95]  Hongming Shan,et al.  Accelerated Correction of Reflection Artifacts by Deep Neural Networks in Photo-Acoustic Tomography , 2019, Applied Sciences.

[96]  Xosé Luís Deán-Ben,et al.  Deep learning optoacoustic tomography with sparse data , 2019, Nature Machine Intelligence.

[97]  S. Arridge,et al.  Quantitative spectroscopic photoacoustic imaging: a review. , 2012, Journal of biomedical optics.

[98]  Tong Tong,et al.  Domain Transform Network for Photoacoustic Tomography from Limited-view and Sparsely Sampled Data , 2020, Photoacoustics.

[99]  Lihong V. Wang,et al.  Second generation optical-resolution photoacoustic microscopy , 2011, BiOS.

[100]  Lihong V. Wang,et al.  Photoacoustic imaging in biomedicine , 2006 .

[101]  Kamal Jnawali,et al.  Automatic cancer tissue detection using multispectral photoacoustic imaging , 2019, International Journal of Computer Assisted Radiology and Surgery.

[102]  Phaneendra K. Yalavarthy,et al.  Sinogram super-resolution and denoising convolutional neural network (SRCN) for limited data photoacoustic tomography , 2020 .

[103]  Feng Gao,et al.  Y-Net: Hybrid deep learning image reconstruction for photoacoustic tomography in vivo , 2020, Photoacoustics.

[104]  Shuang Yu,et al.  Multi-task Neural Networks with Spatial Activation for Retinal Vessel Segmentation and Artery/Vein Classification , 2019, MICCAI.

[105]  Daniel Razansky,et al.  Efficient segmentation of multi-modal optoacoustic and ultrasound images using convolutional neural networks , 2020, BiOS.

[106]  Zheng Zhang,et al.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems , 2015, ArXiv.

[107]  Yong Zhou,et al.  Tutorial on photoacoustic tomography , 2016, Journal of biomedical optics.

[108]  Lena Maier-Hein,et al.  Estimation of blood oxygenation with learned spectral decoloring for quantitative photoacoustic imaging (LSD-qPAI) , 2019, ArXiv.

[109]  Leslie Ying,et al.  A New Deep Learning Network for Mitigating Limited-view and Under-sampling Artifacts in Ring-shaped Photoacoustic Tomography , 2020, Comput. Medical Imaging Graph..

[110]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[111]  Mohammad Mohammadi,et al.  High-quality photoacoustic image reconstruction based on deep convolutional neural network: towards intra-operative photoacoustic imaging , 2020, Biomedical physics & engineering express.

[112]  Fei Gao,et al.  Reconstruct the Photoacoustic Image Based On Deep Learning with Multi-frequency Ring-shape Transducer Array , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[113]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[114]  Andreas Hauptmann,et al.  Towards accurate quantitative photoacoustic imaging: learning vascular blood oxygen saturation in 3D , 2020 .

[115]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[116]  B. Cox,et al.  Photoacoustic tomography in absorbing acoustic media using time reversal , 2010 .

[117]  Manojit Pramanik,et al.  Deep neural network-based bandwidth enhancement of photoacoustic data , 2017, Journal of biomedical optics.

[118]  Yoshiyuki Sankai,et al.  Deep learning-enhanced LED-based photoacoustic imaging , 2020, BiOS.

[119]  Junjie Yao,et al.  Single-impulse Panoramic Photoacoustic Computed Tomography of Small-animal Whole-body Dynamics at High Spatiotemporal Resolution , 2017, Nature Biomedical Engineering.

[120]  Steven Guan,et al.  Fully Dense UNet for 2-D Sparse Photoacoustic Tomography Artifact Removal , 2018, IEEE Journal of Biomedical and Health Informatics.

[121]  Yuqi Tang,et al.  Photoacoustic tomography of blood oxygenation: A mini review , 2018, Photoacoustics.

[122]  Alexandros G. Dimakis,et al.  Deep Learning Techniques for Inverse Problems in Imaging , 2020, IEEE Journal on Selected Areas in Information Theory.

[123]  Lihong V. Wang,et al.  Universal back-projection algorithm for photoacoustic computed tomography. , 2005 .

[124]  Marc Niethammer,et al.  DeepAtlas: Joint Semi-Supervised Learning of Image Registration and Segmentation , 2019, MICCAI.

[125]  Stephan Antholzer,et al.  NETT regularization for compressed sensing photoacoustic tomography , 2019, BiOS.

[126]  Daniel Razansky,et al.  Deep Learning for Automatic Segmentation of Hybrid Optoacoustic Ultrasound (OPUS) Images , 2020, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[127]  Hongming Shan,et al.  Simultaneous reconstruction of the initial pressure and sound speed in photoacoustic tomography using a deep-learning approach , 2019, Optical Engineering + Applications.

[128]  Manojit Pramanik,et al.  Deep Neural Network-Based Sinogram Super-Resolution and Bandwidth Enhancement for Limited-Data Photoacoustic Tomography , 2020, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.