Why batch and user evaluations do not give the same results

A. H. Turpin, William (Bill) Hersh

Research output: Contribution to journalArticle

91 Citations (Scopus)

Abstract

Much system-oriented evaluation of information retrieval systems has used the Cranfield approach based upon queries run against test collections in a batch mode. Some researchers have questioned whether this approach can be applied to the real world, but little data exists for or against that assertion. We have studied this question in the context of the TREC Interactive Track. Previous results demonstrated that improved performance as measured by relevance-based metrics in batch studies did not correspond with the results of outcomes based on real user searching tasks. The experiments in this paper analyzed those results to determine why this occurred. Our assessment showed that while the queries entered by real users into systems yielding better results in batch studies gave comparable gains in ranking of relevant documents for those users, they did not translate into better performance on specific tasks. This was most likely due to users being able to adequately find and utilize relevant documents ranked further down the output list.

Original languageEnglish (US)
Pages (from-to)225-231
Number of pages7
JournalSIGIR Forum (ACM Special Interest Group on Information Retrieval)
StatePublished - 2001
Externally publishedYes

Fingerprint

Information retrieval systems
Experiments
Batch
Evaluation
Query

Keywords

  • Information retrieval evaluation
  • Interactive retrieval
  • Text Retrieval Conference (TREC)

ASJC Scopus subject areas

  • Management Information Systems
  • Hardware and Architecture

Cite this

@article{fb1076b53bab42c79db368d97cde95c9,
title = "Why batch and user evaluations do not give the same results",
abstract = "Much system-oriented evaluation of information retrieval systems has used the Cranfield approach based upon queries run against test collections in a batch mode. Some researchers have questioned whether this approach can be applied to the real world, but little data exists for or against that assertion. We have studied this question in the context of the TREC Interactive Track. Previous results demonstrated that improved performance as measured by relevance-based metrics in batch studies did not correspond with the results of outcomes based on real user searching tasks. The experiments in this paper analyzed those results to determine why this occurred. Our assessment showed that while the queries entered by real users into systems yielding better results in batch studies gave comparable gains in ranking of relevant documents for those users, they did not translate into better performance on specific tasks. This was most likely due to users being able to adequately find and utilize relevant documents ranked further down the output list.",
keywords = "Information retrieval evaluation, Interactive retrieval, Text Retrieval Conference (TREC)",
author = "Turpin, {A. H.} and Hersh, {William (Bill)}",
year = "2001",
language = "English (US)",
pages = "225--231",
journal = "SIGIR Forum (ACM Special Interest Group on Information Retrieval)",
issn = "0163-5840",
publisher = "Association for Computing Machinery (ACM)",

}

TY - JOUR

T1 - Why batch and user evaluations do not give the same results

AU - Turpin, A. H.

AU - Hersh, William (Bill)

PY - 2001

Y1 - 2001

N2 - Much system-oriented evaluation of information retrieval systems has used the Cranfield approach based upon queries run against test collections in a batch mode. Some researchers have questioned whether this approach can be applied to the real world, but little data exists for or against that assertion. We have studied this question in the context of the TREC Interactive Track. Previous results demonstrated that improved performance as measured by relevance-based metrics in batch studies did not correspond with the results of outcomes based on real user searching tasks. The experiments in this paper analyzed those results to determine why this occurred. Our assessment showed that while the queries entered by real users into systems yielding better results in batch studies gave comparable gains in ranking of relevant documents for those users, they did not translate into better performance on specific tasks. This was most likely due to users being able to adequately find and utilize relevant documents ranked further down the output list.

AB - Much system-oriented evaluation of information retrieval systems has used the Cranfield approach based upon queries run against test collections in a batch mode. Some researchers have questioned whether this approach can be applied to the real world, but little data exists for or against that assertion. We have studied this question in the context of the TREC Interactive Track. Previous results demonstrated that improved performance as measured by relevance-based metrics in batch studies did not correspond with the results of outcomes based on real user searching tasks. The experiments in this paper analyzed those results to determine why this occurred. Our assessment showed that while the queries entered by real users into systems yielding better results in batch studies gave comparable gains in ranking of relevant documents for those users, they did not translate into better performance on specific tasks. This was most likely due to users being able to adequately find and utilize relevant documents ranked further down the output list.

KW - Information retrieval evaluation

KW - Interactive retrieval

KW - Text Retrieval Conference (TREC)

UR - http://www.scopus.com/inward/record.url?scp=0034788434&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0034788434&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:0034788434

SP - 225

EP - 231

JO - SIGIR Forum (ACM Special Interest Group on Information Retrieval)

JF - SIGIR Forum (ACM Special Interest Group on Information Retrieval)

SN - 0163-5840

ER -