Statistical Methods for Analyzing Speedup Learning Experiments

Oren Etzioni, Ruth Etzioni

Research output: Contribution to journalArticlepeer-review

17 Scopus citations


Speedup learning systems are typically evaluated by comparing their impact on a problem solver's performance. The impact is measured by running the problem solver, before and after learning, on a sample of problems randomly drawn from some distribution. Often, the experimenter imposes a bound on the CPU time the problem solver is allowed to spend on any individual problem. Segre et al. (1991) argue that the experimenter's choice of time bound can bias the results of the experiment. To address this problem, we present statistical hypothesis tests specifically designed to analyze speedup data and eliminate this bias. We apply the tests to the data reported by Etzioni (1990a) and show that most (but not all) of the speedups observed are statistically significant.

Original languageEnglish (US)
Pages (from-to)333-347
Number of pages15
JournalMachine Learning
Issue number3
StatePublished - Mar 1994
Externally publishedYes


  • experimental methodology
  • explanation-based learning
  • speedup learning
  • statistics

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence


Dive into the research topics of 'Statistical Methods for Analyzing Speedup Learning Experiments'. Together they form a unique fingerprint.

Cite this