Multimodal medical image retrieval: Image categorization to improve search precision

Jayashree Kalpathy-Cramer, William (Bill) Hersh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Citations (Scopus)

Abstract

Effective medical image retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well researched, they are limited by the quality and availability of the annotations associated with the images. Advances in computer vision have led to methods for using the image itself as the search entity. However, the success of purely content-based techniques, when applied to a diverse set of clinical images, has been somewhat limited and these systems have not had much success in the medical domain. On the other hand, as demonstrated in recent years, a combination of text-based and content-based image retrieval techniques can achieve improved retrieval performance if combined effectively. There are many approaches to multimodal retrieval including early and late fusion of weighed results from the different search engines. In this work, we use automatic annotation based on visual attributes to label images as part of the indexing process and the subsequently use these labels to filter or reorder the results during the retrieval process. Labels for medical images can be categorized along three dimensions - imaging modality, anatomical location and image finding or pathology. Our previous research has indicated that the imaging modality is most easily identified using visual techniques whereas the caption or textual annotation frequently contains the finding or pathological information about the image. Thus, it is best to use visual methods to filter the modality and occasionally, anatomy while it is better to use the textual annotation to find the finding of interest. We have created a modality classifier for the weakly labeled images in our collection using a novel approach that combines affinity propagation for the selection of class exemplars, textons and patch-based descriptors as visual features and a NaiveBayes Nearest Neighbor technique for the classification of modality using visual features. We demonstrate significant improvement in precision attained using this technique for the ImageCLEF medical retrieval task 2009 using both our textual runs as well as runs from all participants in 2009.

Original languageEnglish (US)
Title of host publicationMIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval
Pages165-173
Number of pages9
DOIs
StatePublished - 2010
Event2010 ACM SIGMM International Conference on Multimedia Information Retrieval, MIR 2010 - Philadelphia, PA, United States
Duration: Mar 29 2010Mar 31 2010

Other

Other2010 ACM SIGMM International Conference on Multimedia Information Retrieval, MIR 2010
CountryUnited States
CityPhiladelphia, PA
Period3/29/103/31/10

Fingerprint

Image retrieval
Labels
Imaging techniques
Pathology
Search engines
Information retrieval
Computer vision
Classifiers
Fusion reactions
Education
Availability

Keywords

  • Automatic annotation
  • Classification
  • Image retrieval
  • Machine learning

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Information Systems

Cite this

Kalpathy-Cramer, J., & Hersh, W. B. (2010). Multimodal medical image retrieval: Image categorization to improve search precision. In MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval (pp. 165-173) https://doi.org/10.1145/1743384.1743415

Multimodal medical image retrieval : Image categorization to improve search precision. / Kalpathy-Cramer, Jayashree; Hersh, William (Bill).

MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval. 2010. p. 165-173.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kalpathy-Cramer, J & Hersh, WB 2010, Multimodal medical image retrieval: Image categorization to improve search precision. in MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval. pp. 165-173, 2010 ACM SIGMM International Conference on Multimedia Information Retrieval, MIR 2010, Philadelphia, PA, United States, 3/29/10. https://doi.org/10.1145/1743384.1743415
Kalpathy-Cramer J, Hersh WB. Multimodal medical image retrieval: Image categorization to improve search precision. In MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval. 2010. p. 165-173 https://doi.org/10.1145/1743384.1743415
Kalpathy-Cramer, Jayashree ; Hersh, William (Bill). / Multimodal medical image retrieval : Image categorization to improve search precision. MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval. 2010. pp. 165-173
@inproceedings{aa00d4935a4a43eebb71efc0312a2716,
title = "Multimodal medical image retrieval: Image categorization to improve search precision",
abstract = "Effective medical image retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well researched, they are limited by the quality and availability of the annotations associated with the images. Advances in computer vision have led to methods for using the image itself as the search entity. However, the success of purely content-based techniques, when applied to a diverse set of clinical images, has been somewhat limited and these systems have not had much success in the medical domain. On the other hand, as demonstrated in recent years, a combination of text-based and content-based image retrieval techniques can achieve improved retrieval performance if combined effectively. There are many approaches to multimodal retrieval including early and late fusion of weighed results from the different search engines. In this work, we use automatic annotation based on visual attributes to label images as part of the indexing process and the subsequently use these labels to filter or reorder the results during the retrieval process. Labels for medical images can be categorized along three dimensions - imaging modality, anatomical location and image finding or pathology. Our previous research has indicated that the imaging modality is most easily identified using visual techniques whereas the caption or textual annotation frequently contains the finding or pathological information about the image. Thus, it is best to use visual methods to filter the modality and occasionally, anatomy while it is better to use the textual annotation to find the finding of interest. We have created a modality classifier for the weakly labeled images in our collection using a novel approach that combines affinity propagation for the selection of class exemplars, textons and patch-based descriptors as visual features and a NaiveBayes Nearest Neighbor technique for the classification of modality using visual features. We demonstrate significant improvement in precision attained using this technique for the ImageCLEF medical retrieval task 2009 using both our textual runs as well as runs from all participants in 2009.",
keywords = "Automatic annotation, Classification, Image retrieval, Machine learning",
author = "Jayashree Kalpathy-Cramer and Hersh, {William (Bill)}",
year = "2010",
doi = "10.1145/1743384.1743415",
language = "English (US)",
isbn = "9781605588155",
pages = "165--173",
booktitle = "MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval",

}

TY - GEN

T1 - Multimodal medical image retrieval

T2 - Image categorization to improve search precision

AU - Kalpathy-Cramer, Jayashree

AU - Hersh, William (Bill)

PY - 2010

Y1 - 2010

N2 - Effective medical image retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well researched, they are limited by the quality and availability of the annotations associated with the images. Advances in computer vision have led to methods for using the image itself as the search entity. However, the success of purely content-based techniques, when applied to a diverse set of clinical images, has been somewhat limited and these systems have not had much success in the medical domain. On the other hand, as demonstrated in recent years, a combination of text-based and content-based image retrieval techniques can achieve improved retrieval performance if combined effectively. There are many approaches to multimodal retrieval including early and late fusion of weighed results from the different search engines. In this work, we use automatic annotation based on visual attributes to label images as part of the indexing process and the subsequently use these labels to filter or reorder the results during the retrieval process. Labels for medical images can be categorized along three dimensions - imaging modality, anatomical location and image finding or pathology. Our previous research has indicated that the imaging modality is most easily identified using visual techniques whereas the caption or textual annotation frequently contains the finding or pathological information about the image. Thus, it is best to use visual methods to filter the modality and occasionally, anatomy while it is better to use the textual annotation to find the finding of interest. We have created a modality classifier for the weakly labeled images in our collection using a novel approach that combines affinity propagation for the selection of class exemplars, textons and patch-based descriptors as visual features and a NaiveBayes Nearest Neighbor technique for the classification of modality using visual features. We demonstrate significant improvement in precision attained using this technique for the ImageCLEF medical retrieval task 2009 using both our textual runs as well as runs from all participants in 2009.

AB - Effective medical image retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well researched, they are limited by the quality and availability of the annotations associated with the images. Advances in computer vision have led to methods for using the image itself as the search entity. However, the success of purely content-based techniques, when applied to a diverse set of clinical images, has been somewhat limited and these systems have not had much success in the medical domain. On the other hand, as demonstrated in recent years, a combination of text-based and content-based image retrieval techniques can achieve improved retrieval performance if combined effectively. There are many approaches to multimodal retrieval including early and late fusion of weighed results from the different search engines. In this work, we use automatic annotation based on visual attributes to label images as part of the indexing process and the subsequently use these labels to filter or reorder the results during the retrieval process. Labels for medical images can be categorized along three dimensions - imaging modality, anatomical location and image finding or pathology. Our previous research has indicated that the imaging modality is most easily identified using visual techniques whereas the caption or textual annotation frequently contains the finding or pathological information about the image. Thus, it is best to use visual methods to filter the modality and occasionally, anatomy while it is better to use the textual annotation to find the finding of interest. We have created a modality classifier for the weakly labeled images in our collection using a novel approach that combines affinity propagation for the selection of class exemplars, textons and patch-based descriptors as visual features and a NaiveBayes Nearest Neighbor technique for the classification of modality using visual features. We demonstrate significant improvement in precision attained using this technique for the ImageCLEF medical retrieval task 2009 using both our textual runs as well as runs from all participants in 2009.

KW - Automatic annotation

KW - Classification

KW - Image retrieval

KW - Machine learning

UR - http://www.scopus.com/inward/record.url?scp=77952388718&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77952388718&partnerID=8YFLogxK

U2 - 10.1145/1743384.1743415

DO - 10.1145/1743384.1743415

M3 - Conference contribution

AN - SCOPUS:77952388718

SN - 9781605588155

SP - 165

EP - 173

BT - MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval

ER -