TY - JOUR
T1 - TREC-COVID
T2 - Rationale and structure of an information retrieval shared task for COVID-19
AU - Roberts, Kirk
AU - Alam, Tasmeer
AU - Bedrick, Steven
AU - Demner-Fushman, Dina
AU - Lo, Kyle
AU - Soboroff, Ian
AU - Voorhees, Ellen
AU - Wang, Lucy Lu
AU - Hersh, William R.
N1 - Publisher Copyright:
© The Author(s) 2020.
PY - 2020/9/1
Y1 - 2020/9/1
N2 - TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining 9 important basic IR research questions related to pandemic situations. TREC-COVID differs from traditional IR shared task evaluations with special considerations for the expected users, IR modality considerations, topic development, participant requirements, assessment process, relevance criteria, evaluation metrics, iteration process, projected timeline, and the implications of data use as a post-task test collection. This article describes how all these were addressed for the particular requirements of developing IR systems under a pandemic situation. Finally, initial participation numbers are also provided, which demonstrate the tremendous interest the IR community has in this effort.
AB - TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining 9 important basic IR research questions related to pandemic situations. TREC-COVID differs from traditional IR shared task evaluations with special considerations for the expected users, IR modality considerations, topic development, participant requirements, assessment process, relevance criteria, evaluation metrics, iteration process, projected timeline, and the implications of data use as a post-task test collection. This article describes how all these were addressed for the particular requirements of developing IR systems under a pandemic situation. Finally, initial participation numbers are also provided, which demonstrate the tremendous interest the IR community has in this effort.
KW - COVID-19
KW - Information retrieval
KW - Shared task
UR - http://www.scopus.com/inward/record.url?scp=85084617119&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85084617119&partnerID=8YFLogxK
U2 - 10.1093/jamia/ocaa091
DO - 10.1093/jamia/ocaa091
M3 - Article
C2 - 32365190
AN - SCOPUS:85084617119
SN - 1067-5027
VL - 27
SP - 1431
EP - 1436
JO - Journal of the American Medical Informatics Association
JF - Journal of the American Medical Informatics Association
IS - 9
ER -