Towards new measures of information retrieval evaluation

William (Bill) Hersh, Diane Elliot, David H. Hickam, Stephanie L. Wolf, Anna Molnar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

20 Citations (Scopus)

Abstract

All of the methods currently used to evaluate information retrieval (IR) systems have limitations in their ability to measure how well users are able to acquire information. We utilized an approach to assessing information obtained based on the user's ability to answer questions from a short-answer test. Senior medical students took the ten-question test and then searched one of two IR systems on the five questions for which they were least certain of their answer. Our results showed that pre-searching scores on the test were low but that searching yielded a high proportion of answers with both systems. These methods are able to measure information obtained, and will be used in subsequent studies to assess differences among IR systems.

Original languageEnglish (US)
Title of host publicationSIGIR Forum (ACM Special Interest Group on Information Retrieval)
PublisherACM
Pages164-170
Number of pages7
StatePublished - 1995
EventProceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - Seattle, WA, USA
Duration: Jul 9 1995Jul 13 1995

Other

OtherProceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
CitySeattle, WA, USA
Period7/9/957/13/95

Fingerprint

Information retrieval systems
Information retrieval
Students
Evaluation

ASJC Scopus subject areas

  • Management Information Systems
  • Hardware and Architecture

Cite this

Hersh, W. B., Elliot, D., Hickam, D. H., Wolf, S. L., & Molnar, A. (1995). Towards new measures of information retrieval evaluation. In SIGIR Forum (ACM Special Interest Group on Information Retrieval) (pp. 164-170). ACM.

Towards new measures of information retrieval evaluation. / Hersh, William (Bill); Elliot, Diane; Hickam, David H.; Wolf, Stephanie L.; Molnar, Anna.

SIGIR Forum (ACM Special Interest Group on Information Retrieval). ACM, 1995. p. 164-170.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hersh, WB, Elliot, D, Hickam, DH, Wolf, SL & Molnar, A 1995, Towards new measures of information retrieval evaluation. in SIGIR Forum (ACM Special Interest Group on Information Retrieval). ACM, pp. 164-170, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA, 7/9/95.
Hersh WB, Elliot D, Hickam DH, Wolf SL, Molnar A. Towards new measures of information retrieval evaluation. In SIGIR Forum (ACM Special Interest Group on Information Retrieval). ACM. 1995. p. 164-170
Hersh, William (Bill) ; Elliot, Diane ; Hickam, David H. ; Wolf, Stephanie L. ; Molnar, Anna. / Towards new measures of information retrieval evaluation. SIGIR Forum (ACM Special Interest Group on Information Retrieval). ACM, 1995. pp. 164-170
@inproceedings{bb0dd7e9627842a19b6d7bf5781d0645,
title = "Towards new measures of information retrieval evaluation",
abstract = "All of the methods currently used to evaluate information retrieval (IR) systems have limitations in their ability to measure how well users are able to acquire information. We utilized an approach to assessing information obtained based on the user's ability to answer questions from a short-answer test. Senior medical students took the ten-question test and then searched one of two IR systems on the five questions for which they were least certain of their answer. Our results showed that pre-searching scores on the test were low but that searching yielded a high proportion of answers with both systems. These methods are able to measure information obtained, and will be used in subsequent studies to assess differences among IR systems.",
author = "Hersh, {William (Bill)} and Diane Elliot and Hickam, {David H.} and Wolf, {Stephanie L.} and Anna Molnar",
year = "1995",
language = "English (US)",
pages = "164--170",
booktitle = "SIGIR Forum (ACM Special Interest Group on Information Retrieval)",
publisher = "ACM",

}

TY - GEN

T1 - Towards new measures of information retrieval evaluation

AU - Hersh, William (Bill)

AU - Elliot, Diane

AU - Hickam, David H.

AU - Wolf, Stephanie L.

AU - Molnar, Anna

PY - 1995

Y1 - 1995

N2 - All of the methods currently used to evaluate information retrieval (IR) systems have limitations in their ability to measure how well users are able to acquire information. We utilized an approach to assessing information obtained based on the user's ability to answer questions from a short-answer test. Senior medical students took the ten-question test and then searched one of two IR systems on the five questions for which they were least certain of their answer. Our results showed that pre-searching scores on the test were low but that searching yielded a high proportion of answers with both systems. These methods are able to measure information obtained, and will be used in subsequent studies to assess differences among IR systems.

AB - All of the methods currently used to evaluate information retrieval (IR) systems have limitations in their ability to measure how well users are able to acquire information. We utilized an approach to assessing information obtained based on the user's ability to answer questions from a short-answer test. Senior medical students took the ten-question test and then searched one of two IR systems on the five questions for which they were least certain of their answer. Our results showed that pre-searching scores on the test were low but that searching yielded a high proportion of answers with both systems. These methods are able to measure information obtained, and will be used in subsequent studies to assess differences among IR systems.

UR - http://www.scopus.com/inward/record.url?scp=0029202224&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0029202224&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0029202224

SP - 164

EP - 170

BT - SIGIR Forum (ACM Special Interest Group on Information Retrieval)

PB - ACM

ER -