Variation of relevance assessments for medical image retrieval

Henning Müller, Paul Clough, Bill Hersh, Antoine Geissbühler

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Evaluation is crucial for the success of most research domains, and image retrieval is no exception to this. Recently, several benchmarks have been developed for visual information retrieval such as TRECVID, ImageCLEF, and ImagEval to create frameworks for evaluating image retrieval research. An important part of evaluation is the creation of a ground truth or gold standard to evaluate systems against. Much experience has been gained on creating ground truths for textual information retrieval, but for image retrieval these issues require further research. This article will present the process of generating relevance judgements for the medical image retrieval task of ImageCLEF. Many of the problems encountered can be generalised to other image retrieval tasks as well, so the outcome is not limited to the medical domain. Part of the images analysed for relevance were judged by two assessors, and these are analysed with respect to their consistency and potential problems. Our goal is to obtain more information on the ambiguity of the topics developed and generally to keep the variation amongst relevance assessors low. This might partially reduce the subjectivity of system-oriented evaluation, although the evaluation shows that the differences in relevance judgements only have a limited influence on comparative system ranking. A number of outcomes are presented with a goal in mind to create less ambiguous topics for future evaluation campaigns.

Original languageEnglish (US)
Title of host publicationAdaptive Multimedia Retrieval
Subtitle of host publicationUser, Context, and Feedback - 4th International Workshop, AMR 2006, Revised Selected Papers
Pages232-246
Number of pages15
StatePublished - Dec 1 2007
Event4th International Workshop on Adaptive Multimedia Retrieval, AMR 2006 - Geneva, Switzerland
Duration: Jul 27 2006Jul 28 2006

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume4398 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other4th International Workshop on Adaptive Multimedia Retrieval, AMR 2006
CountrySwitzerland
CityGeneva
Period7/27/067/28/06

    Fingerprint

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Müller, H., Clough, P., Hersh, B., & Geissbühler, A. (2007). Variation of relevance assessments for medical image retrieval. In Adaptive Multimedia Retrieval: User, Context, and Feedback - 4th International Workshop, AMR 2006, Revised Selected Papers (pp. 232-246). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 4398 LNCS).