Abstract
Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.
| Original language | English (US) |
|---|---|
| Pages (from-to) | 17-24 |
| Number of pages | 8 |
| Journal | SIGIR Forum (ACM Special Interest Group on Information Retrieval) |
| DOIs | |
| State | Published - Jan 1 2000 |
| Event | Proceedings of the 23rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000) - Athens, Greece Duration: Jul 24 2000 → Jul 28 2000 |
ASJC Scopus subject areas
- Management Information Systems
- Hardware and Architecture
Fingerprint Dive into the research topics of 'Do batch and user evaluations give the same results?'. Together they form a unique fingerprint.
Cite this
Hersh, W., Turpin, A., Price, S., Chan, B., Kraemer, D., Sacherek, L., & Olson, D. (2000). Do batch and user evaluations give the same results? SIGIR Forum (ACM Special Interest Group on Information Retrieval), 17-24. https://doi.org/10.1145/345508.345539