Catastrophic forgetting in simple networks: an analysis of the pseudorehearsal solution.

Catastrophic forgetting is a major problem for sequential learning in neural networks. One very general solution to this problem, known as 'pseudorehearsal', works well in practice for nonlinear networks but has not been analysed before. This paper formalizes pseudorehearsal in linear networks. We show that the method can fail in low dimensions but is guaranteed to succeed in high dimensions under fairly general conditions. In this case an optimal version of the method is equivalent to a simple modification of the 'delta rule'.

[1]  Robert M. French,et al.  Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionist Networks , 1991 .

[2]  R Ratcliff,et al.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. , 1990, Psychological review.

[3]  Stanislas Dehaene,et al.  Networks of Formal Neurons and Memory Palimpsests , 1986 .

[4]  Francis Crick,et al.  The function of dream sleep , 1983, Nature.

[5]  Robert M. French,et al.  Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks , 1992 .

[6]  S. Lewandowsky,et al.  Catastrophic interference in neural networks , 1995 .

[7]  J. Leo van Hemmen,et al.  Universality of unlearning , 1994, Neural Networks.

[8]  George A. Christos,et al.  Investigation of the Crick-Mitchison reverse-learning dream sleep hypothesis in a dynamical setting , 1996, Neural Networks.

[9]  Anthony V. Robins,et al.  Catastrophic Forgetting, Rehearsal and Pseudorehearsal , 1995, Connect. Sci..

[10]  Robert M. French,et al.  Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity-Stability' Dilemma , 1997, Connect. Sci..

[11]  J. J. Hopfield,et al.  ‘Unlearning’ has a stabilizing effect in collective memories , 1983, Nature.

[12]  M. A. Moore,et al.  Neural network models of list learning , 1991 .

[13]  Jaap M. J. Murre.  Hypertransfer in Neural Networks , 1996, Connect. Sci..

[14]  Anthony V. Robins,et al.  Consolidation in Neural Networks and in the Sleeping Brain , 1996, Connect. Sci..

[15]  Noel E. Sharkey,et al.  An Analysis of Catastrophic Interference , 1995, Connect. Sci..

[16]  J. L. van Hemmen,et al.  Hebbian learning, its correlation catastrophe, and unlearning , 1997 .

[17]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[18]  Anthony V. Robins,et al.  Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks , 1998, Connect. Sci..

[19]  Jacob M.J. Murre,et al.  Learning and Categorization in Modular Neural Networks , 1992 .