SOINN+, a Self-Organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams

Abstract The goal of continuous learning is to acquire and fine-tune knowledge incrementally without erasing already existing knowledge. How to mitigate this erasure, known as catastrophic forgetting, is a grand challenge for machine learning, specifically when systems are trained on evolving data streams. Self-organizing incremental neural networks (SOINN) are a class of neural networks that are designed for continuous learning from non-stationary data. Here, we propose a novel method, SOINN+, which is fundamentally different from the previous generations of SOINN with respect to how “forgetting” is modeled. To achieve a more graceful forgetting, we developed three new concepts: idle time, trustworthiness, and unutility of a node. SOINN+ learns a topology-preserving mapping of the input data to a network structure, which reveals clusters of arbitrary shapes in streams of noisy data. Our experiments with synthetic and real-world data sets showed that SOINN+ can maintain discovered structures even under sudden and recurring concept drifts.

[1]  Christopher Summerfield,et al.  Comparing continual task learning in minds and machines , 2018, Proceedings of the National Academy of Sciences.

[2]  Ira Assent,et al.  Self-Adaptive Anytime Stream Clustering , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[3]  Shen Furao,et al.  An enhanced self-organizing incremental neural network for online unsupervised learning , 2007, Neural Networks.

[4]  Bernd Fritzke,et al.  A Growing Neural Gas Network Learns Topologies , 1994, NIPS.

[5]  João Gama,et al.  Data Stream Classification Guided by Clustering on Nonstationary Environments and Extreme Verification Latency , 2015, SDM.

[6]  William Nick Street,et al.  A streaming ensemble algorithm (SEA) for large-scale classification , 2001, KDD '01.

[7]  Geoff Holmes,et al.  MOA: Massive Online Analysis , 2010, J. Mach. Learn. Res..

[8]  Shen Furao,et al.  An Online Semi-Supervised Clustering Algorithm Based on a Self-organizing Incremental Neural Network , 2007, 2007 International Joint Conference on Neural Networks.

[9]  Shen Furao,et al.  An incremental network for on-line unsupervised classification and topology learning , 2006, Neural Networks.

[10]  Stefan Wermter,et al.  Self-organizing neural integration of pose-motion features for human action recognition , 2015, Front. Neurorobot..

[11]  Stefan Wermter,et al.  Continual Lifelong Learning with Neural Networks: A Review , 2019, Neural Networks.

[12]  Christian Sohler,et al.  StreamKM++: A clustering algorithm for data streams , 2010, JEAL.

[13]  Stefan Wermter,et al.  Lifelong learning of human actions with deep neural network self-organization , 2017, Neural Networks.

[14]  Anthony V. Robins,et al.  Catastrophic Forgetting, Rehearsal and Pseudorehearsal , 1995, Connect. Sci..

[15]  Mustapha Lebbah,et al.  A new Growing Neural Gas for clustering data streams , 2016, Neural Networks.

[16]  Derek Hoiem,et al.  Learning without Forgetting , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Hui Yu,et al.  An incremental online semi-supervised active learning algorithm based on self-organizing incremental neural network , 2011, Neural Computing and Applications.

[18]  Osamu Hasegawa,et al.  Associative Memory for Online Learning in Noisy Environments Using Self-Organizing Incremental Neural Network , 2009, IEEE Transactions on Neural Networks.

[19]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[20]  Stephen R. Marsland,et al.  A self-organising network that grows when required , 2002, Neural Networks.

[21]  Osamu Hasegawa,et al.  Nonparametric Density Estimation Based on Self-Organizing Incremental Neural Network for Large Noisy Data , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Mohamed Medhat Gaber,et al.  EACD: evolutionary adaptation to concept drifts in data streams , 2019, Data Mining and Knowledge Discovery.

[23]  R. French,et al.  Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions , 1994 .

[24]  Martin Schrimpf,et al.  Continual Learning with Self-Organizing Maps , 2019, ArXiv.

[25]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[26]  Fred Henrik Hamker,et al.  Life-long learning Cell Structures--continuously learning without catastrophic interference , 2001, Neural Networks.

[27]  Razvan Pascanu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[28]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[29]  P. Rousseeuw,et al.  Alternatives to the Median Absolute Deviation , 1993 .

[30]  Philip S. Yu,et al.  A Framework for Clustering Evolving Data Streams , 2003, VLDB.

[31]  Gregory Ditzler,et al.  Learning in Nonstationary Environments: A Survey , 2015, IEEE Computational Intelligence Magazine.

[32]  Osamu Hasegawa,et al.  Developmental Word Acquisition and Grammar Learning by Humanoid Robots Through a Self-Organizing Incremental Neural Network , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[33]  Aram Kawewong,et al.  Reasoning on the Self-Organizing Incremental Associative Memory for Online Robot Path Planning , 2010, IEICE Trans. Inf. Syst..

[34]  Ying Wah Teh,et al.  On Density-Based Data Streams Clustering Algorithms: A Survey , 2014, Journal of Computer Science and Technology.

[35]  Michael S. C. Thomas,et al.  Critical periods and catastrophic interference effects in the development of self-organizing feature maps. , 2008, Developmental science.

[36]  Bernd Fritzke,et al.  Growing cell structures--A self-organizing network for unsupervised and supervised learning , 1994, Neural Networks.

[37]  Davide Maltoni,et al.  CORe50: a New Dataset and Benchmark for Continuous Object Recognition , 2017, CoRL.

[38]  Daniel Berrar,et al.  Self-Organizing Incremental Neural Networks for Continual Learning , 2019, IJCAI.

[39]  Stefan Wermter,et al.  Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization , 2018, Front. Neurorobot..

[40]  Shen Furao,et al.  A fast nearest neighbor classifier based on self-organizing incremental neural network , 2008, Neural Networks.

[41]  Toyoaki Nishida,et al.  Incremental learning of gestures for human–robot interaction , 2009, AI & SOCIETY.

[42]  Gerald Sommer,et al.  Dynamic Cell Structure Learns Perfectly Topology Preserving Map , 1995, Neural Computation.

[43]  Michael D. Howard,et al.  Complementary Learning Systems , 2014, Cogn. Sci..

[44]  Xiong Xiao,et al.  A Load-Balancing Self-Organizing Incremental Neural Network , 2014, IEEE Transactions on Neural Networks and Learning Systems.