Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems

Phoebe M. Roberts, Aaron M. Cohen, William R. Hersh

Research output: Contribution to journalArticle

18 Scopus citations

Abstract

With the help of a team of expert biologist judges, the TREC Genomics track has generated four large sets of "gold standard" test collections, comprised of over a hundred unique topics, two kinds of ad hoc retrieval tasks, and their corresponding relevance judgments. Over the years of the track, increasingly complex tasks necessitated the creation of judging tools and training guidelines to accommodate teams of part-time short-term workers from a variety of specialized biological scientific backgrounds, and to address consistency and reproducibility of the assessment process. Important lessons were learned about factors that influenced the utility of the test collections including topic design, annotations provided by judges, methods used for identifying and training judges, and providing a central moderator "meta-judge".

Original languageEnglish (US)
Pages (from-to)81-97
Number of pages17
JournalInformation Retrieval
Volume12
Issue number1
DOIs
StatePublished - Feb 1 2009

    Fingerprint

Keywords

  • Evaluation
  • Information retrieval
  • Inter-annotator agreement
  • Reference standards
  • Text mining

ASJC Scopus subject areas

  • Information Systems
  • Library and Information Sciences

Cite this