A fast histogram-based postprocessor that improves posterior probability estimates

Wei Wei, Todd K. Leen, Etienne Barnard

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Although the outputs of neural network classifiers are often considered to be estimates of posterior class probabilities, the literature that assesses the calibration accuracy of these estimates illustrates that practical networks often fall far short of being ideal estimators. The theorems used to justify treating network outputs as good posterior estimates are based on several assumptions: that the network is sufficiently complex to model the posterior distribution accurately, that there are sufficient training data to specify the network, and that the optimization routine is capable of finding the global minimum of the cost function. Any or all of these assumptions may be violated in practice. This article does three things. First, we apply a simple, previously used histogram technique to assess graphically the accuracy of posterior estimates with respect to individual classes. Second, we introduce a simple and fast remapping procedure that transforms network outputs to provide better estimates of posteriors. Third, we use the remapping in a real-world telephone speech recognition system. The remapping results in a 10% reduction of both word-level error rates (from 4.53% to 4.06%) and sentence-level error rates (from 16.38% to 14.69%) on one corpus, and a 29% reduction at sentence-level error (from 6.3% to 4.5%) on another. The remapping required negligible additional overhead (in terms of both parameters and calculations). McNemar's test shows that these levels of improvement are statistically significant.

Original languageEnglish (US)
Pages (from-to)1235-1248
Number of pages14
JournalNeural Computation
Volume11
Issue number5
StatePublished - Jul 1 1999

Fingerprint

Telephone
Calibration
Costs and Cost Analysis
Speech recognition
Cost functions
Classifiers
Neural networks

ASJC Scopus subject areas

  • Artificial Intelligence
  • Control and Systems Engineering
  • Neuroscience(all)

Cite this

A fast histogram-based postprocessor that improves posterior probability estimates. / Wei, Wei; Leen, Todd K.; Barnard, Etienne.

In: Neural Computation, Vol. 11, No. 5, 01.07.1999, p. 1235-1248.

Research output: Contribution to journalArticle

Wei, W, Leen, TK & Barnard, E 1999, 'A fast histogram-based postprocessor that improves posterior probability estimates', Neural Computation, vol. 11, no. 5, pp. 1235-1248.
Wei, Wei ; Leen, Todd K. ; Barnard, Etienne. / A fast histogram-based postprocessor that improves posterior probability estimates. In: Neural Computation. 1999 ; Vol. 11, No. 5. pp. 1235-1248.
@article{e11e567df474437c963afed12b6299b4,
title = "A fast histogram-based postprocessor that improves posterior probability estimates",
abstract = "Although the outputs of neural network classifiers are often considered to be estimates of posterior class probabilities, the literature that assesses the calibration accuracy of these estimates illustrates that practical networks often fall far short of being ideal estimators. The theorems used to justify treating network outputs as good posterior estimates are based on several assumptions: that the network is sufficiently complex to model the posterior distribution accurately, that there are sufficient training data to specify the network, and that the optimization routine is capable of finding the global minimum of the cost function. Any or all of these assumptions may be violated in practice. This article does three things. First, we apply a simple, previously used histogram technique to assess graphically the accuracy of posterior estimates with respect to individual classes. Second, we introduce a simple and fast remapping procedure that transforms network outputs to provide better estimates of posteriors. Third, we use the remapping in a real-world telephone speech recognition system. The remapping results in a 10{\%} reduction of both word-level error rates (from 4.53{\%} to 4.06{\%}) and sentence-level error rates (from 16.38{\%} to 14.69{\%}) on one corpus, and a 29{\%} reduction at sentence-level error (from 6.3{\%} to 4.5{\%}) on another. The remapping required negligible additional overhead (in terms of both parameters and calculations). McNemar's test shows that these levels of improvement are statistically significant.",
author = "Wei Wei and Leen, {Todd K.} and Etienne Barnard",
year = "1999",
month = "7",
day = "1",
language = "English (US)",
volume = "11",
pages = "1235--1248",
journal = "Neural Computation",
issn = "0899-7667",
publisher = "MIT Press Journals",
number = "5",

}

TY - JOUR

T1 - A fast histogram-based postprocessor that improves posterior probability estimates

AU - Wei, Wei

AU - Leen, Todd K.

AU - Barnard, Etienne

PY - 1999/7/1

Y1 - 1999/7/1

N2 - Although the outputs of neural network classifiers are often considered to be estimates of posterior class probabilities, the literature that assesses the calibration accuracy of these estimates illustrates that practical networks often fall far short of being ideal estimators. The theorems used to justify treating network outputs as good posterior estimates are based on several assumptions: that the network is sufficiently complex to model the posterior distribution accurately, that there are sufficient training data to specify the network, and that the optimization routine is capable of finding the global minimum of the cost function. Any or all of these assumptions may be violated in practice. This article does three things. First, we apply a simple, previously used histogram technique to assess graphically the accuracy of posterior estimates with respect to individual classes. Second, we introduce a simple and fast remapping procedure that transforms network outputs to provide better estimates of posteriors. Third, we use the remapping in a real-world telephone speech recognition system. The remapping results in a 10% reduction of both word-level error rates (from 4.53% to 4.06%) and sentence-level error rates (from 16.38% to 14.69%) on one corpus, and a 29% reduction at sentence-level error (from 6.3% to 4.5%) on another. The remapping required negligible additional overhead (in terms of both parameters and calculations). McNemar's test shows that these levels of improvement are statistically significant.

AB - Although the outputs of neural network classifiers are often considered to be estimates of posterior class probabilities, the literature that assesses the calibration accuracy of these estimates illustrates that practical networks often fall far short of being ideal estimators. The theorems used to justify treating network outputs as good posterior estimates are based on several assumptions: that the network is sufficiently complex to model the posterior distribution accurately, that there are sufficient training data to specify the network, and that the optimization routine is capable of finding the global minimum of the cost function. Any or all of these assumptions may be violated in practice. This article does three things. First, we apply a simple, previously used histogram technique to assess graphically the accuracy of posterior estimates with respect to individual classes. Second, we introduce a simple and fast remapping procedure that transforms network outputs to provide better estimates of posteriors. Third, we use the remapping in a real-world telephone speech recognition system. The remapping results in a 10% reduction of both word-level error rates (from 4.53% to 4.06%) and sentence-level error rates (from 16.38% to 14.69%) on one corpus, and a 29% reduction at sentence-level error (from 6.3% to 4.5%) on another. The remapping required negligible additional overhead (in terms of both parameters and calculations). McNemar's test shows that these levels of improvement are statistically significant.

UR - http://www.scopus.com/inward/record.url?scp=0040907586&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0040907586&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:0040907586

VL - 11

SP - 1235

EP - 1248

JO - Neural Computation

JF - Neural Computation

SN - 0899-7667

IS - 5

ER -