Statistical Methods for Analyzing Speedup Learning Experiments

Oren Etzioni, Ruth Etzioni

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

Speedup learning systems are typically evaluated by comparing their impact on a problem solver's performance. The impact is measured by running the problem solver, before and after learning, on a sample of problems randomly drawn from some distribution. Often, the experimenter imposes a bound on the CPU time the problem solver is allowed to spend on any individual problem. Segre et al. (1991) argue that the experimenter's choice of time bound can bias the results of the experiment. To address this problem, we present statistical hypothesis tests specifically designed to analyze speedup data and eliminate this bias. We apply the tests to the data reported by Etzioni (1990a) and show that most (but not all) of the speedups observed are statistically significant.

Original languageEnglish (US)
Pages (from-to)333-347
Number of pages15
JournalMachine Learning
Volume14
Issue number3
DOIs
StatePublished - Jan 1 1994
Externally publishedYes

Fingerprint

Statistical tests
Program processors
Learning systems
Statistical methods
Experiments

Keywords

  • experimental methodology
  • explanation-based learning
  • speedup learning
  • statistics

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Cite this

Statistical Methods for Analyzing Speedup Learning Experiments. / Etzioni, Oren; Etzioni, Ruth.

In: Machine Learning, Vol. 14, No. 3, 01.01.1994, p. 333-347.

Research output: Contribution to journalArticle

Etzioni, Oren ; Etzioni, Ruth. / Statistical Methods for Analyzing Speedup Learning Experiments. In: Machine Learning. 1994 ; Vol. 14, No. 3. pp. 333-347.
@article{531c449cb7214cd4b318b0bbcb0242ff,
title = "Statistical Methods for Analyzing Speedup Learning Experiments",
abstract = "Speedup learning systems are typically evaluated by comparing their impact on a problem solver's performance. The impact is measured by running the problem solver, before and after learning, on a sample of problems randomly drawn from some distribution. Often, the experimenter imposes a bound on the CPU time the problem solver is allowed to spend on any individual problem. Segre et al. (1991) argue that the experimenter's choice of time bound can bias the results of the experiment. To address this problem, we present statistical hypothesis tests specifically designed to analyze speedup data and eliminate this bias. We apply the tests to the data reported by Etzioni (1990a) and show that most (but not all) of the speedups observed are statistically significant.",
keywords = "experimental methodology, explanation-based learning, speedup learning, statistics",
author = "Oren Etzioni and Ruth Etzioni",
year = "1994",
month = "1",
day = "1",
doi = "10.1023/A:1022617931401",
language = "English (US)",
volume = "14",
pages = "333--347",
journal = "Machine Learning",
issn = "0885-6125",
publisher = "Springer Netherlands",
number = "3",

}

TY - JOUR

T1 - Statistical Methods for Analyzing Speedup Learning Experiments

AU - Etzioni, Oren

AU - Etzioni, Ruth

PY - 1994/1/1

Y1 - 1994/1/1

N2 - Speedup learning systems are typically evaluated by comparing their impact on a problem solver's performance. The impact is measured by running the problem solver, before and after learning, on a sample of problems randomly drawn from some distribution. Often, the experimenter imposes a bound on the CPU time the problem solver is allowed to spend on any individual problem. Segre et al. (1991) argue that the experimenter's choice of time bound can bias the results of the experiment. To address this problem, we present statistical hypothesis tests specifically designed to analyze speedup data and eliminate this bias. We apply the tests to the data reported by Etzioni (1990a) and show that most (but not all) of the speedups observed are statistically significant.

AB - Speedup learning systems are typically evaluated by comparing their impact on a problem solver's performance. The impact is measured by running the problem solver, before and after learning, on a sample of problems randomly drawn from some distribution. Often, the experimenter imposes a bound on the CPU time the problem solver is allowed to spend on any individual problem. Segre et al. (1991) argue that the experimenter's choice of time bound can bias the results of the experiment. To address this problem, we present statistical hypothesis tests specifically designed to analyze speedup data and eliminate this bias. We apply the tests to the data reported by Etzioni (1990a) and show that most (but not all) of the speedups observed are statistically significant.

KW - experimental methodology

KW - explanation-based learning

KW - speedup learning

KW - statistics

UR - http://www.scopus.com/inward/record.url?scp=2542423444&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=2542423444&partnerID=8YFLogxK

U2 - 10.1023/A:1022617931401

DO - 10.1023/A:1022617931401

M3 - Article

AN - SCOPUS:2542423444

VL - 14

SP - 333

EP - 347

JO - Machine Learning

JF - Machine Learning

SN - 0885-6125

IS - 3

ER -