Multi-task learning for ultrasound image formation and segmentation directly from raw in vivo data

Deep neural networks have demonstrated potential to both create images and segment structures of interest directly from raw ultrasound data in one step, through an end-to-end transformation. Building on previous work from our group, subaperture beamformed IQ data from in vivo breast cyst data was the input to a custom network that outputs parallel B-mode and cyst segmentation images. Our new model includes bright point and line targets during training to overcome the limited field of view challenges encountered with our previous deep learning models, which were purely trained using simulations of cysts and homogeneous tissue structures. This new network resulted in cyst contrast values of $-33.07\pm 10.79\ \text{dB}, -32.09\pm 0.04 \text{dB}$, and $-15.95\pm 12.04\ \text{dB}$ for simulated, phantom, and in vivo data, respectively, which is an improvement over the contrast of corresponding delay and sum (DAS) images (i.e., $-17.37\pm 6.06\ \text{dB},\ -17.14\pm 0.16\ \text{dB}$, and $14.80\pm 1.30\ \text{dB}$ for simulated, phantom, and in vivo, respectively). Higher dice similarity coefficients (DSCs) were obtained with in vivo data with the new network (.83 \pm 0.01$) when compared to our previous model (.63\pm 0.03$), and fewer false positives were encountered. This work demonstrates the feasibility of using multi-task learning to simultaneously form a B-mode image and cyst segmentation with a wider field of view that is appropriate for in vivo breast imaging. These results have promising implications for multiple tasks, including emphasizing or de-emphasizing structures of interest for diagnostic, interventional, automated, and semi-automated decision making.