Using media fusion and domain dimensions to improve precision in medical image retrieval

Saïd Radhouani, Jayashree Kalpathy-Cramer, Steven Bedrick, Brian Bakke, William (Bill) Hersh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Citations (Scopus)

Abstract

In this paper, we focus on improving retrieval performance, especially early precision, in the task of solving medical multimodal queries. The queries we deal with consist of a visual component, given as a set of image-examples, and textual annotation, provided as a set of words. The queries' semantic content can be classified along three domain dimensions: anatomy, pathology, and modality. To solve these queries, we interpret their semantic content using both textual and visual data. Medical images often are accompanied by textual annotations, which in turn typically include explicit mention of their image's anatomy or pathology. Annotations rarely include explicit mention of image modality, however. To address this, we use an image's visual features to identify its modality. Our system thereby performs image retrieval by combining purely visual information about an image with information derived from its textual annotations. In order to experimentally evaluate our approach, we performed a set of experiments using the 2009 ImageCLEFmed collection using our integrated system as well as a purely textual retrieval system. Our integrated approach consistently outperformed our text-only system by 43% in MAP and by 71% in precision within the top 5 retrieved documents. We conclude that this improved performance is due to our method of combining visual and textual features.

Original languageEnglish (US)
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages223-230
Number of pages8
Volume6242 LNCS
DOIs
StatePublished - 2010
Event10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009 - Corfu, Greece
Duration: Sep 30 2009Oct 2 2009

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume6242 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009
CountryGreece
CityCorfu
Period9/30/0910/2/09

Fingerprint

Image retrieval
Pathology
Image Retrieval
Medical Image
Fusion
Fusion reactions
Semantics
Annotation
Query
Modality
Anatomy
Retrieval
Integrated System
Experiments
Vision
Evaluate
Experiment

Keywords

  • Domain Dimensions
  • Image Classification
  • Image Modality Extraction
  • Media Fusion
  • Medical Image Retrieval
  • Performance Evaluation

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Radhouani, S., Kalpathy-Cramer, J., Bedrick, S., Bakke, B., & Hersh, W. B. (2010). Using media fusion and domain dimensions to improve precision in medical image retrieval. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 6242 LNCS, pp. 223-230). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6242 LNCS). https://doi.org/10.1007/978-3-642-15751-6_27

Using media fusion and domain dimensions to improve precision in medical image retrieval. / Radhouani, Saïd; Kalpathy-Cramer, Jayashree; Bedrick, Steven; Bakke, Brian; Hersh, William (Bill).

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 6242 LNCS 2010. p. 223-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6242 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Radhouani, S, Kalpathy-Cramer, J, Bedrick, S, Bakke, B & Hersh, WB 2010, Using media fusion and domain dimensions to improve precision in medical image retrieval. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 6242 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 6242 LNCS, pp. 223-230, 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, 9/30/09. https://doi.org/10.1007/978-3-642-15751-6_27
Radhouani S, Kalpathy-Cramer J, Bedrick S, Bakke B, Hersh WB. Using media fusion and domain dimensions to improve precision in medical image retrieval. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 6242 LNCS. 2010. p. 223-230. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-15751-6_27
Radhouani, Saïd ; Kalpathy-Cramer, Jayashree ; Bedrick, Steven ; Bakke, Brian ; Hersh, William (Bill). / Using media fusion and domain dimensions to improve precision in medical image retrieval. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 6242 LNCS 2010. pp. 223-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{aaec138a536545559598e734c3b0b1aa,
title = "Using media fusion and domain dimensions to improve precision in medical image retrieval",
abstract = "In this paper, we focus on improving retrieval performance, especially early precision, in the task of solving medical multimodal queries. The queries we deal with consist of a visual component, given as a set of image-examples, and textual annotation, provided as a set of words. The queries' semantic content can be classified along three domain dimensions: anatomy, pathology, and modality. To solve these queries, we interpret their semantic content using both textual and visual data. Medical images often are accompanied by textual annotations, which in turn typically include explicit mention of their image's anatomy or pathology. Annotations rarely include explicit mention of image modality, however. To address this, we use an image's visual features to identify its modality. Our system thereby performs image retrieval by combining purely visual information about an image with information derived from its textual annotations. In order to experimentally evaluate our approach, we performed a set of experiments using the 2009 ImageCLEFmed collection using our integrated system as well as a purely textual retrieval system. Our integrated approach consistently outperformed our text-only system by 43{\%} in MAP and by 71{\%} in precision within the top 5 retrieved documents. We conclude that this improved performance is due to our method of combining visual and textual features.",
keywords = "Domain Dimensions, Image Classification, Image Modality Extraction, Media Fusion, Medical Image Retrieval, Performance Evaluation",
author = "Sa{\"i}d Radhouani and Jayashree Kalpathy-Cramer and Steven Bedrick and Brian Bakke and Hersh, {William (Bill)}",
year = "2010",
doi = "10.1007/978-3-642-15751-6_27",
language = "English (US)",
isbn = "3642157505",
volume = "6242 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "223--230",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - Using media fusion and domain dimensions to improve precision in medical image retrieval

AU - Radhouani, Saïd

AU - Kalpathy-Cramer, Jayashree

AU - Bedrick, Steven

AU - Bakke, Brian

AU - Hersh, William (Bill)

PY - 2010

Y1 - 2010

N2 - In this paper, we focus on improving retrieval performance, especially early precision, in the task of solving medical multimodal queries. The queries we deal with consist of a visual component, given as a set of image-examples, and textual annotation, provided as a set of words. The queries' semantic content can be classified along three domain dimensions: anatomy, pathology, and modality. To solve these queries, we interpret their semantic content using both textual and visual data. Medical images often are accompanied by textual annotations, which in turn typically include explicit mention of their image's anatomy or pathology. Annotations rarely include explicit mention of image modality, however. To address this, we use an image's visual features to identify its modality. Our system thereby performs image retrieval by combining purely visual information about an image with information derived from its textual annotations. In order to experimentally evaluate our approach, we performed a set of experiments using the 2009 ImageCLEFmed collection using our integrated system as well as a purely textual retrieval system. Our integrated approach consistently outperformed our text-only system by 43% in MAP and by 71% in precision within the top 5 retrieved documents. We conclude that this improved performance is due to our method of combining visual and textual features.

AB - In this paper, we focus on improving retrieval performance, especially early precision, in the task of solving medical multimodal queries. The queries we deal with consist of a visual component, given as a set of image-examples, and textual annotation, provided as a set of words. The queries' semantic content can be classified along three domain dimensions: anatomy, pathology, and modality. To solve these queries, we interpret their semantic content using both textual and visual data. Medical images often are accompanied by textual annotations, which in turn typically include explicit mention of their image's anatomy or pathology. Annotations rarely include explicit mention of image modality, however. To address this, we use an image's visual features to identify its modality. Our system thereby performs image retrieval by combining purely visual information about an image with information derived from its textual annotations. In order to experimentally evaluate our approach, we performed a set of experiments using the 2009 ImageCLEFmed collection using our integrated system as well as a purely textual retrieval system. Our integrated approach consistently outperformed our text-only system by 43% in MAP and by 71% in precision within the top 5 retrieved documents. We conclude that this improved performance is due to our method of combining visual and textual features.

KW - Domain Dimensions

KW - Image Classification

KW - Image Modality Extraction

KW - Media Fusion

KW - Medical Image Retrieval

KW - Performance Evaluation

UR - http://www.scopus.com/inward/record.url?scp=78049328603&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=78049328603&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-15751-6_27

DO - 10.1007/978-3-642-15751-6_27

M3 - Conference contribution

AN - SCOPUS:78049328603

SN - 3642157505

SN - 9783642157509

VL - 6242 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 223

EP - 230

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -