Automatic labeling of continuous wave Doppler images based on combined image and sentence networks

As medical imaging datasets grow, we are approaching the era of big data for radiologic decision support systems. This requires renewed efforts in dataset curation and labeling. We propose a methodology for weak labeling of medical images for attributes such as anatomy and disease that relies on image to sentence transformation. The methodology consists of three models, a convolutional neural network that is trained on a coarse classification task and acts as an image feature generator, a language model to map sentences to a fixed length space, and a multi-layer perceptron that acts as a function approximator to map images to the sentence space. The transform model is trained on matched image-sentence pairs on a dataset of echocardiography studies. For a given image, labels are extracted from the closest sentences to the output of the image-sentence transform. We show that the resulting solution has an 78.2% accuracy in labeling Doppler images with aortic stenosis. We also show that the retrieved sentences are consistent with the true sentences in terms of meaning with an average BLEU score of 0.34, matching the current highly performing machine translation solutions.