论文信息 - Annealed RNN Learning of Finite State Automata

Annealed RNN Learning of Finite State Automata

In recurrent neural network (RNN) learning of finite state automata (FSA), we discuss how a neuro gain (β) influences the stability of the state representation and the performance of the learning. We formally show that the existence of the critical neuro gain (β0): any β larger than β0 makes an RNN maintain the stable representation of states of an acquired FSA. Considering the existence of β0 and avoidance of local minima, we propose a new RNN learning method with the scheduling of β, called an annealed RNN learning. Our experiments show that the annealed RNN learning went beyond than a constant β learning.

Ryohei Nakano | Ken-ichi Arai

[1] Srimat T. Chakradhar,et al. First-order versus second-order single-layer recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[2] C. Lee Giles,et al. Using Prior Knowledge in a {NNPDA} to Learn Context-Free Languages , 1992, NIPS.

[3] Anders Krogh,et al. Introduction to the theory of neural computation , 1994, The advanced book program.

[4] Panagiotis Manolios,et al. First-Order Recurrent Neural Networks and Deterministic Finite State Automata , 1994, Neural Computation.

[5] Geoffrey E. Hinton,et al. Learning internal representations by error propagation , 1986 .

[6] Naonori Ueda,et al. Deterministic Annealing Variant of the EM Algorithm , 1994, NIPS.

[7] L. K. Li,et al. Fixed point analysis for discrete-time recurrent neural networks , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.