A Structured Learning Approach for Medical Image Indexing and Retrieval

Medical images are critical assets for medical diagnosis, research, and teaching. To facilitate automatic indexing and retrieval of large medical image databases, we propose a structured framework for designing and learning vocabularies of meaningful medical terms with associated visual appearance from image samples. These VisMed terms span a new feature space to represent medical image contents. After a multi-scale detection process, a medical image is indexed as compact spatial distributions of VisMed terms. When queries are in the form of example images, both a query image and a database image can be matched based on their distributions of VisMed terms, much like the matching of feature-based histograms though the bins refer to semantic medical terms. In addition, a flexible tiling (FlexiTile) matching scheme has been proposed to compare the similarity between two medical images of arbitrary aspect ratios. This matching scheme supports similarity-based retrieval with visual queries. The ranked list of such retrieval is denoted as “i2r-vk-sim.txt” in our submission to ImageCLEF 2005. When a query is expressed as a text description that involves modality, anatomy, and pathology etc, it can be translated into a visual query representation that chains the presences of VisMed terms with spatial significance via logical operators (AND, OR, NOT) and spatial quantifiers for automatic query processing based on the VisMed image indexes. This query formulation and processing scheme allows semantics-based retrieval with text queries. The ranked list of such retrieval is denoted as “i2r-vksem.txt” in our submission to ImageCLEF 2005. By fusing the ranked lists from both the similarity-based and semantics-based retrievals, we can leverage on the information expressed in both visual and text queries respectively. The ranked list of such retrieval is denoted as “i2r-vk-avg.txt” in our submission to ImageCLEF 2005. We apply the VisMed approach on the Medical Image Retrieval task of the ImageCLEF track under CLEF 2005. Based on 0.3% (i.e. 158 images) of the 50, 026 images from 4 collections plus 96 images obtained from the web, we cropped 1460 image regions to train and validate 39 VisMed terms using support vector machines. The Mean Average Precisions (MAP) over 25 query topics for the submissions “i2r-vk-sim.txt”, “i2rvk-sem.txt”, and “i2r-vk-avg.txt” are 0.0721, 0.06, and 0.0921 respectively, according to the evaluation results released by the ImageCLEF 2005 organizers. The submission “i2r-vk-avg.txt” is also combined with text-only submissions “IPALI2R Tn” and “IPALI2R T” to form submissions for mixed retrieval. The best MAP among these submissions for mixed retrieval is 0.2821 from submission “IPALI2R TIan”.