Relevance and retrieval evaluation: Perspectives from medicine

Research output: Contribution to journalArticle

45 Citations (Scopus)

Abstract

The traditional notion of topical relevance has allowed much useful work to be done in the evaluation of retrieval systems, but has limitations for complete assessment of retrieval systems. While topical relevance can be effective in evaluating various indexing and retrieval approaches, it is ineffective for measuring the impact that systems have on users. An alternative is to use a more situational definition of relevance, which takes account of the impact of the system on the user. Both types of relevance are examined from the standpoint of the medical domain, concluding that each have their appropriate use. But in medicine there is increasing emphasis on outcomes‐oriented research which, when applied to information science, requires that the impact of an information system on the activities which prompt its use be assessed. An iterative model of retrieval evaluation is proposed, starting first with the use of topical relevance to insure documents on the subject can be retrieved. This is followed by the use of situational relevance to show the user can interact positively with the system. The final step is to study how the system impacts the user in the purpose for which the system was consulted, which can be done by methods such as protocol analysis and simulation. These diverse types of studies are necessary to increase our understanding of the nature of retrieval systems. © 1994 John Wiley & Sons, Inc.

Original languageEnglish (US)
Pages (from-to)201-206
Number of pages6
JournalJournal of the American Society for Information Science
Volume45
Issue number3
DOIs
StatePublished - 1994

Fingerprint

Information science
Medicine
Information systems
medicine
evaluation
Evaluation
indexing
information science
information system
simulation

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Relevance and retrieval evaluation : Perspectives from medicine. / Hersh, William (Bill).

In: Journal of the American Society for Information Science, Vol. 45, No. 3, 1994, p. 201-206.

Research output: Contribution to journalArticle

@article{ae039dd87a634de2a36d0d574fc5e0a3,
title = "Relevance and retrieval evaluation: Perspectives from medicine",
abstract = "The traditional notion of topical relevance has allowed much useful work to be done in the evaluation of retrieval systems, but has limitations for complete assessment of retrieval systems. While topical relevance can be effective in evaluating various indexing and retrieval approaches, it is ineffective for measuring the impact that systems have on users. An alternative is to use a more situational definition of relevance, which takes account of the impact of the system on the user. Both types of relevance are examined from the standpoint of the medical domain, concluding that each have their appropriate use. But in medicine there is increasing emphasis on outcomes‐oriented research which, when applied to information science, requires that the impact of an information system on the activities which prompt its use be assessed. An iterative model of retrieval evaluation is proposed, starting first with the use of topical relevance to insure documents on the subject can be retrieved. This is followed by the use of situational relevance to show the user can interact positively with the system. The final step is to study how the system impacts the user in the purpose for which the system was consulted, which can be done by methods such as protocol analysis and simulation. These diverse types of studies are necessary to increase our understanding of the nature of retrieval systems. {\circledC} 1994 John Wiley & Sons, Inc.",
author = "Hersh, {William (Bill)}",
year = "1994",
doi = "10.1002/(SICI)1097-4571(199404)45:3<201::AID-ASI9>3.0.CO;2-W",
language = "English (US)",
volume = "45",
pages = "201--206",
journal = "Journal of the Association for Information Science and Technology",
issn = "2330-1635",
publisher = "John Wiley and Sons Ltd",
number = "3",

}

TY - JOUR

T1 - Relevance and retrieval evaluation

T2 - Perspectives from medicine

AU - Hersh, William (Bill)

PY - 1994

Y1 - 1994

N2 - The traditional notion of topical relevance has allowed much useful work to be done in the evaluation of retrieval systems, but has limitations for complete assessment of retrieval systems. While topical relevance can be effective in evaluating various indexing and retrieval approaches, it is ineffective for measuring the impact that systems have on users. An alternative is to use a more situational definition of relevance, which takes account of the impact of the system on the user. Both types of relevance are examined from the standpoint of the medical domain, concluding that each have their appropriate use. But in medicine there is increasing emphasis on outcomes‐oriented research which, when applied to information science, requires that the impact of an information system on the activities which prompt its use be assessed. An iterative model of retrieval evaluation is proposed, starting first with the use of topical relevance to insure documents on the subject can be retrieved. This is followed by the use of situational relevance to show the user can interact positively with the system. The final step is to study how the system impacts the user in the purpose for which the system was consulted, which can be done by methods such as protocol analysis and simulation. These diverse types of studies are necessary to increase our understanding of the nature of retrieval systems. © 1994 John Wiley & Sons, Inc.

AB - The traditional notion of topical relevance has allowed much useful work to be done in the evaluation of retrieval systems, but has limitations for complete assessment of retrieval systems. While topical relevance can be effective in evaluating various indexing and retrieval approaches, it is ineffective for measuring the impact that systems have on users. An alternative is to use a more situational definition of relevance, which takes account of the impact of the system on the user. Both types of relevance are examined from the standpoint of the medical domain, concluding that each have their appropriate use. But in medicine there is increasing emphasis on outcomes‐oriented research which, when applied to information science, requires that the impact of an information system on the activities which prompt its use be assessed. An iterative model of retrieval evaluation is proposed, starting first with the use of topical relevance to insure documents on the subject can be retrieved. This is followed by the use of situational relevance to show the user can interact positively with the system. The final step is to study how the system impacts the user in the purpose for which the system was consulted, which can be done by methods such as protocol analysis and simulation. These diverse types of studies are necessary to increase our understanding of the nature of retrieval systems. © 1994 John Wiley & Sons, Inc.

UR - http://www.scopus.com/inward/record.url?scp=84989554840&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84989554840&partnerID=8YFLogxK

U2 - 10.1002/(SICI)1097-4571(199404)45:3<201::AID-ASI9>3.0.CO;2-W

DO - 10.1002/(SICI)1097-4571(199404)45:3<201::AID-ASI9>3.0.CO;2-W

M3 - Article

AN - SCOPUS:84989554840

VL - 45

SP - 201

EP - 206

JO - Journal of the Association for Information Science and Technology

JF - Journal of the Association for Information Science and Technology

SN - 2330-1635

IS - 3

ER -