Neural networks with small weights implement finite memory machines

Recent experimental studies indicate that recurrent networks initialized with ‘small’ weights are inherently biased towards finite memory machines [37]. This paper establishes a theoretical counterpart: we prove that recurrent networks with small weights or contractive transition function, respectively, can be approximated arbitrarily well on input sequences of unbounded length by a finite memory machine. Conversely, every finite memory machine can be simulated by a recurrent network with contractive transition function. Hence initialization with small weights induces an architectural bias into learning with recurrent neural networks. This bias has benefits from the point of view of statistical learning theory: it emphasizes regions of the weight space where good generalization can be expected. It is well known that standard recurrent neural networks are not distribution independent learnable in the PAC sense. We prove that recurrent networks with contractive transition function with a fixed parameter of the contraction fulfill the so-called distribution independent UCED property and hence are distribution independent PAC-learnable unlike general recurrent networks.