Constrained Second-Order Recurrent Networks for Finite-State Automata Induction

This paper presents an improved training algorithm for second-order dynamical recurrent networks applied to the problem of finite-state automata (FSA) induction. Second-order networks allow for a natural encoding of finite-state automata in which each second-order connection weight corresponds to one transition in a finite-state automaton. In practice, however, when trained using gradient descent, these networks almost never assume this type of encoding and sophisticated algorithms must be used to extract the encoded automata. This paper suggests a simple modification to the standard error function for second-order dynamical recurrent networks which encourages these networks to assume natural FSA encodings when trained using gradient descent. This obviates the need for cluster-based extraction techniques and provides a simple method for guaranteeing the stability of the network for arbitrarily long sequences. Initial results also suggest that fewer training strings must be presented to achieve convergence using the modified error.