A performance and failure analysis of SAPHIRE with a MEDLINE test collection

William (Bill) Hersh, David H. Hickam, R. Brian Haynes, K. Ann Mckibbon

Research output: Contribution to journalArticle

51 Citations (Scopus)

Abstract

Objective: Assess the performance of the SAPHIRE automated information retrieval system. Design: Comparative study of automated and human searching of a MEDLINE test collection. Measurements: Recall and precision of SAPHIRE were compared with those attributes of novice physicians, expert physicians, and librarians for a test collection of 75 queries and 2, 334 citations. Failure analysis assessed the efficacy of the Metathesaurus as a concept vocabulary; the reasons for retrieval of nonrelevant articles and nonretrieval of relevant articles; and the effect of changing the weighting formula for relevance ranking of retrieved articles. Results: Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. Conclusion: The current version of the Metathesaurus, as utilized by SAPHIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries. The most likely cause for retrieval of nonrelevant articles was the presence of some or all of the search terms in the article, with frequencies high enough to lead to retrieval. The most likely cause for non-retrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hierarchically related terms present instead. There were significant variations in performance when SAPHIRE's concept-weighing formulas were modified.

Original languageEnglish (US)
Pages (from-to)51-60
Number of pages10
JournalJournal of the American Medical Informatics Association
Volume1
Issue number1
StatePublished - Jan 1994

Fingerprint

MEDLINE
Unified Medical Language System
Librarians
Physicians
Vocabulary
Information Systems

ASJC Scopus subject areas

  • Health Informatics
  • Medicine(all)

Cite this

A performance and failure analysis of SAPHIRE with a MEDLINE test collection. / Hersh, William (Bill); Hickam, David H.; Haynes, R. Brian; Mckibbon, K. Ann.

In: Journal of the American Medical Informatics Association, Vol. 1, No. 1, 01.1994, p. 51-60.

Research output: Contribution to journalArticle

@article{2e60aace8ce24dd6977c71e12b570ac2,
title = "A performance and failure analysis of SAPHIRE with a MEDLINE test collection",
abstract = "Objective: Assess the performance of the SAPHIRE automated information retrieval system. Design: Comparative study of automated and human searching of a MEDLINE test collection. Measurements: Recall and precision of SAPHIRE were compared with those attributes of novice physicians, expert physicians, and librarians for a test collection of 75 queries and 2, 334 citations. Failure analysis assessed the efficacy of the Metathesaurus as a concept vocabulary; the reasons for retrieval of nonrelevant articles and nonretrieval of relevant articles; and the effect of changing the weighting formula for relevance ranking of retrieved articles. Results: Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. Conclusion: The current version of the Metathesaurus, as utilized by SAPHIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries. The most likely cause for retrieval of nonrelevant articles was the presence of some or all of the search terms in the article, with frequencies high enough to lead to retrieval. The most likely cause for non-retrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hierarchically related terms present instead. There were significant variations in performance when SAPHIRE's concept-weighing formulas were modified.",
author = "Hersh, {William (Bill)} and Hickam, {David H.} and Haynes, {R. Brian} and Mckibbon, {K. Ann}",
year = "1994",
month = "1",
language = "English (US)",
volume = "1",
pages = "51--60",
journal = "Journal of the American Medical Informatics Association",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "1",

}

TY - JOUR

T1 - A performance and failure analysis of SAPHIRE with a MEDLINE test collection

AU - Hersh, William (Bill)

AU - Hickam, David H.

AU - Haynes, R. Brian

AU - Mckibbon, K. Ann

PY - 1994/1

Y1 - 1994/1

N2 - Objective: Assess the performance of the SAPHIRE automated information retrieval system. Design: Comparative study of automated and human searching of a MEDLINE test collection. Measurements: Recall and precision of SAPHIRE were compared with those attributes of novice physicians, expert physicians, and librarians for a test collection of 75 queries and 2, 334 citations. Failure analysis assessed the efficacy of the Metathesaurus as a concept vocabulary; the reasons for retrieval of nonrelevant articles and nonretrieval of relevant articles; and the effect of changing the weighting formula for relevance ranking of retrieved articles. Results: Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. Conclusion: The current version of the Metathesaurus, as utilized by SAPHIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries. The most likely cause for retrieval of nonrelevant articles was the presence of some or all of the search terms in the article, with frequencies high enough to lead to retrieval. The most likely cause for non-retrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hierarchically related terms present instead. There were significant variations in performance when SAPHIRE's concept-weighing formulas were modified.

AB - Objective: Assess the performance of the SAPHIRE automated information retrieval system. Design: Comparative study of automated and human searching of a MEDLINE test collection. Measurements: Recall and precision of SAPHIRE were compared with those attributes of novice physicians, expert physicians, and librarians for a test collection of 75 queries and 2, 334 citations. Failure analysis assessed the efficacy of the Metathesaurus as a concept vocabulary; the reasons for retrieval of nonrelevant articles and nonretrieval of relevant articles; and the effect of changing the weighting formula for relevance ranking of retrieved articles. Results: Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. Conclusion: The current version of the Metathesaurus, as utilized by SAPHIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries. The most likely cause for retrieval of nonrelevant articles was the presence of some or all of the search terms in the article, with frequencies high enough to lead to retrieval. The most likely cause for non-retrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hierarchically related terms present instead. There were significant variations in performance when SAPHIRE's concept-weighing formulas were modified.

UR - http://www.scopus.com/inward/record.url?scp=0028367731&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0028367731&partnerID=8YFLogxK

M3 - Article

C2 - 7719787

AN - SCOPUS:0028367731

VL - 1

SP - 51

EP - 60

JO - Journal of the American Medical Informatics Association

JF - Journal of the American Medical Informatics Association

SN - 1067-5027

IS - 1

ER -