Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks

Pseudorehearsal is a mechanism proposed by Robins which alleviates catastrophic forgetting in multi-layer perceptron networks. In this paper, we extend the exploration of pseudorehearsal to a Hopfield-type net. The same general principles apply: old information can be rehearsed if it is available, and if it is not available, then generating and rehearsing approximations of old information that 'map' the behaviour of the network can also be effective at preserving the actual old information itself. The details of the pseudorehearsal mechanism, however, benefit from being adapted to the dynamics of Hopfield nets so as to exploit the extra attractors created in state space during learning. These attractors are usually described as 'spurious' or 'cross-talk', and regarded as undesirable, interfering with the retention of the trained population items. Our simulations have shown that, in another sense, such attractors can in fact be useful in preserving the learned population. In general terms, a solution to th...

[1]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[2]  J. L. van Hemmen,et al.  Hebbian learning, its correlation catastrophe, and unlearning , 1997 .

[3]  Neil Davey,et al.  High performance associative memory models and symmetric connections , 2000 .

[4]  C. Smith,et al.  Increases in number of REMS and REM density in humans following an intensive learning period. , 1991, Sleep.

[5]  James L. McClelland,et al.  A Hippocampal Model of Recognition Memory , 1997, NIPS.

[6]  Maxwell B. Stinchcombe,et al.  Neural network approximation of continuous functionals and continuous functions on compactifications , 1999, Neural Networks.

[7]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[8]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[9]  Edward E. Smith,et al.  Concepts and concept formation. , 1984, Annual review of psychology.

[10]  R. French.  Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference , 2019, Proceedings of the Sixteenth Annual Conference of the Cognitive Science Society.

[11]  Matthias Löwe.  On the storage capacity of Hopfield models with correlated patterns , 1998 .

[12]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[13]  A. Bovier.  Sharp upper bounds on perfect retrieval in the Hopfield model , 1999 .

[14]  E. Tulving.  Multiple memory systems and consciousness. , 1987, Human neurobiology.

[15]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[16]  J. J. Hopfield,et al.  ‘Unlearning’ has a stabilizing effect in collective memories , 1983, Nature.

[17]  Carlyle T. Smith,et al.  Impaired motor memory for a pursuit rotor task following Stage 2 sleep loss in college students , 1994, Journal of sleep research.

[18]  C. Pearlman,et al.  Latent learning impaired by REM sleep deprivation , 1971 .

[19]  Chandan Dasgupta,et al.  Retrieval Properties of a Hopfield Model with Random Asymmetric Interactions , 2000, Neural Computation.

[20]  Teuvo Kohonen,et al.  Content-addressable memories , 1980 .

[21]  Anthony V. Robins,et al.  Consolidation in Neural Networks and in the Sleeping Brain , 1996, Connect. Sci..

[22]  T. Bliss,et al.  Long‐lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path , 1973, The Journal of physiology.

[23]  E. Gardner.  The space of interactions in neural network models , 1988 .

[24]  Christian Hölscher,et al.  Neuronal mechanisms of memory formation : concepts of long-term potentiation and beyond , 2000 .

[25]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[26]  R Ratcliff,et al.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. , 1990, Psychological review.

[27]  George A. Christos,et al.  Investigation of the Crick-Mitchison reverse-learning dream sleep hypothesis in a dynamical setting , 1996, Neural Networks.

[28]  R. Nicoll,et al.  Long-term potentiation--a decade of progress? , 1999, Science.

[29]  Nicolas Brunel,et al.  Learning internal representations in an attractor neural network with analogue neurons , 1995 .

[30]  Hideyuki Koyama,et al.  Analysis of the Recalling Processes of Associative Memory by an Integral Representation of the Sign Function , 1996, Neural Networks.

[31]  Halbert White,et al.  Artificial Neural Networks: Approximation and Learning Theory , 1992 .

[32]  Bernard Ans,et al.  Avoiding catastrophic forgetting by coupling two reverberating neural networks , 1997 .

[33]  D. J. Wallace,et al.  Training with noise and the storage of correlated patterns in a neural network model , 1989 .

[34]  K. J. Schultz,et al.  Fully parallel integrated CAM/RAM using preclassification to enable large capacities , 1996, IEEE J. Solid State Circuits.

[35]  Anthony V. Robins,et al.  Catastrophic Forgetting, Rehearsal and Pseudorehearsal , 1995, Connect. Sci..

[36]  K. McRae,et al.  Catastrophic Interference is Eliminated in Pretrained Networks , 1993 .

[37]  Alex Pentland,et al.  Analysis of Neural Networks with Redundancy , 1990, Neural Computation.

[38]  Axel Cleeremans,et al.  Experience-dependent changes in cerebral activation during human REM sleep , 2000, Nature Neuroscience.

[39]  M. W. Brown,et al.  Recognition memory: neuronal substrates of the judgement of prior occurrence , 1998, Progress in Neurobiology.

[40]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[41]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[42]  B. Hars,et al.  Processing of learned information in paradoxical sleep: relevance for memory , 1995, Behavioural Brain Research.

[43]  C. Pearlman,et al.  REM sleep and information processing: Evidence from animal studies , 1979, Neuroscience & Biobehavioral Reviews.

[44]  Robert E. Mercer,et al.  The Parallel Transfer of Task Knowledge Using Dynamic Learning Rates Based on a Measure of Relatedness , 1998, Learning to Learn.

[45]  Boes,et al.  Statistical mechanics for networks of graded-response neurons. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[46]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[47]  Ted Abel,et al.  Transgenic approaches to cognition , 1995, Current Opinion in Neurobiology.

[48]  Francis Crick,et al.  The function of dream sleep , 1983, Nature.

[49]  Stanislas Dehaene,et al.  Networks of Formal Neurons and Memory Palimpsests , 1986 .

[50]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1952, The Journal of physiology.

[51]  Jacob M.J. Murre,et al.  Learning and Categorization in Modular Neural Networks , 1992 .

[52]  Geoffrey E. Hinton,et al.  Learning representations of back-propagation errors , 1986 .

[53]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[54]  Carlyle Smith,et al.  Sleep states, memory processes and synaptic plasticity , 1996, Behavioural Brain Research.

[55]  Matthias Löwe.  On the Storage Capacity of the Hopfield Model with Biased Patterns , 1999, IEEE Trans. Inf. Theory.

[56]  M. Fishbein,et al.  Sleep, Dreams and Memory , 1979 .

[57]  S. Blomfield.  Arithmetical operations performed by nerve cells. , 1974, Brain research.

[58]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[59]  F. Dudek,et al.  Intracellular electrophysiological study of suprachiasmatic nucleus neurons in rodents: inhibitory synaptic mechanisms. , 1991, The Journal of physiology.

[60]  Amos J. Storkey,et al.  The basins of attraction of a new Hopfield learning rule , 1999, Neural Networks.

[61]  Robert M. French,et al.  Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity-Stability' Dilemma , 1997, Connect. Sci..

[62]  Francis Crick,et al.  REM sleep and neural nets , 1995, Behavioural Brain Research.

[63]  R. French.  Catastrophic forgetting in connectionist networks , 1999, Trends in Cognitive Sciences.

[64]  Marcus R. Frean,et al.  A "Thermal" Perceptron Learning Rule , 1992, Neural Computation.

[65]  E. Kandel,et al.  Genetic Demonstration of a Role for PKA in the Late Phase of LTP and in Hippocampus-Based Long-Term Memory , 1997, Cell.

[66]  S. Lewandowsky,et al.  Catastrophic interference in neural networks , 1995 .

[67]  R. Durrett.  Probability: Theory and Examples , 1993 .

[68]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[69]  Masato Okada,et al.  Notions of Associative Memory and Sparse Coding , 1996, Neural Networks.

[70]  J. Murre.  TraceLink: A model of amnesia and consolidation of memory , 1996, Hippocampus.

[71]  A. G. Hanlon.  Content-Addressable and Associative Memory Systems a Survey , 1966, IEEE Trans. Electron. Comput..

[72]  E Ruppin,et al.  Neuronal regulation versus synaptic unlearning in memory maintenance mechanisms. , 1998, Network.

[73]  Amos J. Storkey,et al.  Increasing the Capacity of a Hopfield Network without Sacrificing Functionality , 1997, ICANN.

[74]  E. Gardner,et al.  Maximum Storage Capacity in Neural Networks , 1987 .

[75]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[76]  Robert M. French,et al.  Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks , 1992 .

[77]  P Alvarez,et al.  Memory consolidation and the medial temporal lobe: a simple network model. , 1994, Proceedings of the National Academy of Sciences of the United States of America.

[78]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[79]  George Marsaglia,et al.  Toward a universal random number generator , 1987 .

[80]  Albert B Novikoff,et al.  ON CONVERGENCE PROOFS FOR PERCEPTRONS , 1963 .

[81]  C. Pearlman,et al.  REM sleep deprivation impairs bar-press acquisition in rats. , 1974, Physiology & behavior.

[82]  Jean Ruiz,et al.  Memory capacity in neural networks with spatial correlations between attractors , 1999 .

[83]  T. Morita,et al.  Exactly solvable model of a spin glass , 1976 .

[84]  Anthony V. Robins,et al.  The consolidation of learning during sleep: comparing the pseudorehearsal and unlearning accounts , 1999, Neural Networks.

[85]  J. Jonas,et al.  Human optic nerve fiber count and optic disc size. , 1992, Investigative ophthalmology & visual science.

[86]  L. Nadel,et al.  Multiple trace theory of human memory: Computational, neuroimaging, and neuropsychological results , 2000, Hippocampus.

[87]  Marcus Frean,et al.  Catastrophic forgetting in simple networks: an analysis of the pseudorehearsal solution. , 1999 .

[88]  Opper,et al.  Learning of correlated patterns in spin-glass networks by local learning rules. , 1987, Physical review letters.

[89]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[90]  Richard C. Atkinson,et al.  Human Memory: A Proposed System and its Control Processes , 1968, Psychology of Learning and Motivation.

[91]  C. Pearlman,et al.  Brief posttrial REM sleep deprivation impairs discrimination learning in rats , 1973 .

[92]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[93]  F. N. Dempster,et al.  Interference and inhibition in cognition: An historical perspective , 1995 .

[94]  Carlyle T. Smith,et al.  Sleep states and memory processes , 1995, Behavioural Brain Research.

[95]  Michael Recce,et al.  A model of hippocampal function , 1994, Neural Networks.

[96]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[97]  Eytan Ruppin,et al.  Memory Maintenance via Neuronal Regulation , 1998, Neural Computation.

[98]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[99]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[100]  Noel E. Sharkey,et al.  An Analysis of Catastrophic Interference , 1995, Connect. Sci..

[101]  M. A. Moore,et al.  Neural network models of list learning , 1991 .

[102]  Friedrich T. Sommer,et al.  Associative Data Storage and Retrieval in Neural Networks , 1996 .

[103]  A CarpenterGail,et al.  The ART of Adaptive Pattern Recognition by a Self-Organizing Neural Network , 1988 .

[104]  T. Bliss,et al.  Long‐lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path , 1973, The Journal of physiology.

[105]  D Marr,et al.  A computational theory of human stereo vision. , 1979, Proceedings of the Royal Society of London. Series B, Biological sciences.