TREC-COVID: Rationale and structure of an information retrieval shared task for COVID-19

Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, William R. Hersh

Research output: Contribution to journalArticlepeer-review

78 Scopus citations

Abstract

TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining 9 important basic IR research questions related to pandemic situations. TREC-COVID differs from traditional IR shared task evaluations with special considerations for the expected users, IR modality considerations, topic development, participant requirements, assessment process, relevance criteria, evaluation metrics, iteration process, projected timeline, and the implications of data use as a post-task test collection. This article describes how all these were addressed for the particular requirements of developing IR systems under a pandemic situation. Finally, initial participation numbers are also provided, which demonstrate the tremendous interest the IR community has in this effort.

Original languageEnglish (US)
Pages (from-to)1431-1436
Number of pages6
JournalJournal of the American Medical Informatics Association
Volume27
Issue number9
DOIs
StatePublished - Sep 1 2020

Keywords

  • COVID-19
  • Information retrieval
  • Shared task

ASJC Scopus subject areas

  • Health Informatics

Fingerprint

Dive into the research topics of 'TREC-COVID: Rationale and structure of an information retrieval shared task for COVID-19'. Together they form a unique fingerprint.

Cite this