Multimodal medical image retrieval: Image categorization to improve search precision

Jayashree Kalpathy-Cramer, William (Bill) Hersh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Scopus citations

Abstract

Effective medical image retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well researched, they are limited by the quality and availability of the annotations associated with the images. Advances in computer vision have led to methods for using the image itself as the search entity. However, the success of purely content-based techniques, when applied to a diverse set of clinical images, has been somewhat limited and these systems have not had much success in the medical domain. On the other hand, as demonstrated in recent years, a combination of text-based and content-based image retrieval techniques can achieve improved retrieval performance if combined effectively. There are many approaches to multimodal retrieval including early and late fusion of weighed results from the different search engines. In this work, we use automatic annotation based on visual attributes to label images as part of the indexing process and the subsequently use these labels to filter or reorder the results during the retrieval process. Labels for medical images can be categorized along three dimensions - imaging modality, anatomical location and image finding or pathology. Our previous research has indicated that the imaging modality is most easily identified using visual techniques whereas the caption or textual annotation frequently contains the finding or pathological information about the image. Thus, it is best to use visual methods to filter the modality and occasionally, anatomy while it is better to use the textual annotation to find the finding of interest. We have created a modality classifier for the weakly labeled images in our collection using a novel approach that combines affinity propagation for the selection of class exemplars, textons and patch-based descriptors as visual features and a NaiveBayes Nearest Neighbor technique for the classification of modality using visual features. We demonstrate significant improvement in precision attained using this technique for the ImageCLEF medical retrieval task 2009 using both our textual runs as well as runs from all participants in 2009.

Original languageEnglish (US)
Title of host publicationMIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval
Pages165-173
Number of pages9
DOIs
Publication statusPublished - 2010
Event2010 ACM SIGMM International Conference on Multimedia Information Retrieval, MIR 2010 - Philadelphia, PA, United States
Duration: Mar 29 2010Mar 31 2010

Other

Other2010 ACM SIGMM International Conference on Multimedia Information Retrieval, MIR 2010
CountryUnited States
CityPhiladelphia, PA
Period3/29/103/31/10

    Fingerprint

Keywords

  • Automatic annotation
  • Classification
  • Image retrieval
  • Machine learning

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Information Systems

Cite this

Kalpathy-Cramer, J., & Hersh, W. B. (2010). Multimodal medical image retrieval: Image categorization to improve search precision. In MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval (pp. 165-173) https://doi.org/10.1145/1743384.1743415