Mapping visual features to semantic profiles for retrieval in medical imaging

Content based image retrieval is highly relevant in medical imaging, since it makes vast amounts of imaging data accessible for comparison during diagnosis. Finding image similarity measures that reflect diagnostically relevant relationships is challenging, since the overall appearance variability is high compared to often subtle signatures of diseases. To learn models that capture the relationship between semantic clinical information and image elements at scale, we have to rely on data generated during clinical routine (images and radiology reports), since expert annotation is prohibitively costly. Here we show that re-mapping visual features extracted from medical imaging data based on weak labels that can be found in corresponding radiology reports creates descriptions of local image content capturing clinically relevant information. We show that these semantic profiles enable higher recall and precision during retrieval compared to visual features, and that we can even map semantic terms describing clinical findings from radiology reports to localized image volume areas.

[1]  Horst Bischof,et al.  MIForests: Multiple-Instance Learning with Randomized Trees , 2010, ECCV.

[2]  Jacob D. Furst,et al.  Content-based image retrieval for pulmonary computed tomography nodule images , 2007, SPIE Medical Imaging.

[3]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Zhi-Hua Zhou,et al.  Learning a distance metric from multi-instance multi-label data , 2009, CVPR.

[5]  Cordelia Schmid,et al.  Multiple Instance Metric Learning from Automatically Labeled Bags of Faces , 2010, ECCV.

[6]  Adrien Depeursinge,et al.  Affine-invariant texture analysis and retrieval of 3D medical images with clinical context integration , 2010 .

[7]  Sudipta Mukhopadhyay,et al.  Content-based image retrieval for interstitial lung diseases , 2012, 2012 IEEE International Conference on Signal Processing, Computing and Control.

[8]  Vincent Lepetit,et al.  Fast Keypoint Recognition Using Random Ferns , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Markus Holzer,et al.  Over-Segmentation of 3D Medical Image Volumes based on Monogenic Cues , 2014 .

[10]  Thomas Hofmann,et al.  Multiple instance learning with generalized support vector machines , 2002, AAAI/IAAI.

[11]  Roberto Cipolla,et al.  Semantic texton forests for image categorization and segmentation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Ye Xu,et al.  MDCT-based 3-D texture classification of emphysema and early smoking related lung pathologies , 2006, IEEE Transactions on Medical Imaging.

[13]  R. Robb,et al.  The Lung Tissue Research Consortium: An extensive open database containing histological, clinical, and radiological data to study chronic lung disease , 2006, The Insight Journal.

[14]  Alexander C. Berg,et al.  Automatic Attribute Discovery and Characterization from Noisy Web Data , 2010, ECCV.

[15]  Carla E. Brodley,et al.  Unsupervised Feature Selection Applied to Content-Based Retrieval of Lung Images , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Nassir Navab,et al.  Building Implicit Dictionaries Based on Extreme Random Clustering for Modality Recognition , 2011, MCBR-CDS.

[17]  Tao Mei,et al.  Concurrent Multiple Instance Learning for Image Categorization , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Vincent Lepetit,et al.  Fast Keypoint Recognition in Ten Lines of Code , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Henrik Boström,et al.  Estimating class probabilities in random forests , 2007, Sixth International Conference on Machine Learning and Applications (ICMLA 2007).

[20]  M.,et al.  Statistical and Structural Approaches to Texture , 2022 .

[21]  Tao Mei,et al.  Joint multi-label multi-instance learning for image classification , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Dimitri Van De Ville,et al.  Near-Affine-Invariant Texture Learning for Lung Tissue Analysis Using Isotropic Wavelet Frames , 2012, IEEE Transactions on Information Technology in Biomedicine.

[23]  Franz Kainberger,et al.  Texture Bags: Anomaly Retrieval in Medical Images Based on Local 3D-Texture Similarity , 2011, MCBR-CDS.

[24]  C. Langlotz.  RadLex: a new method for indexing online educational materials. , 2006, Radiographics : a review publication of the Radiological Society of North America, Inc.

[25]  Loong Fah Cheong,et al.  Randomized Locality Sensitive Vocabularies for Bag-of-Features Model , 2010, ECCV.

[26]  Pedro M. Domingos,et al.  Tree Induction for Probability-Based Ranking , 2003, Machine Learning.