Using curvature information for fast stochastic search

Genevieve B. Orr, Todd K. Leen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

19 Scopus citations

Abstract

We present an algorithm for fast stochastic gradient descent that uses a nonlinear adaptive momentum scheme to optimize the late time convergence rate. The algorithm makes effective use of curvature information, requires only O(n) storage and computation, and delivers convergence rates close to the theoretical optimum. We demonstrate the technique on linear and large nonlinear back-prop networks.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems
PublisherNeural information processing systems foundation
Pages606-612
Number of pages7
ISBN (Print)0262100657, 9780262100656
Publication statusPublished - 1997
Event10th Annual Conference on Neural Information Processing Systems, NIPS 1996 - Denver, CO, United States
Duration: Dec 2 1996Dec 5 1996

Other

Other10th Annual Conference on Neural Information Processing Systems, NIPS 1996
CountryUnited States
CityDenver, CO
Period12/2/9612/5/96

    Fingerprint

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Orr, G. B., & Leen, T. K. (1997). Using curvature information for fast stochastic search. In Advances in Neural Information Processing Systems (pp. 606-612). Neural information processing systems foundation.