Optimal asymptotic learning rate: Macroscopic versus microscopic dynamics

We investigate the asymptotic dynamics of on-line learning for neural networks, and provide an exact solution to the network dynamics at late times under various annealing schedules. The dynamics is solved using two different frameworks: the master equation and order parameter dynamics, which concentrate on microscopic and macroscopic parameters, respectively. The two approaches provide complementary descriptions of the dynamics. Optimal annealing rates and the corresponding prefactors are derived for soft committee machine networks with hidden layers of arbitrary size.