Machine Learning Using Cellular Automata Based Feature Expansion and Reservoir Computing

In this paper, we introduce a novel framework of cellular automata based computing that is capable of long short-term memory. Cellular automaton is used as the reservoir of dynamical systems. Input is randomly projected onto the initial conditions of automaton cells and nonlinear computation is performed on the input via application of a rule in the automaton for a period of time. The evolution of the automaton creates a space-time volume of the automaton state space, and it is used as the feature vector. The proposed framework requires orders of magnitude less computation compared to Echo State Networks. We prove that cellular automaton reservoir holds a distributed representation of attribute statistics, which provides a more effective computation than local representation. It is possible to estimate the kernel for linear cellular automata via metric learning, that enables a much more efficient distance computation in support vector machines framework.

[1]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[2]  Özgür Yilmaz,et al.  Classification of Occluded Objects Using Fast Recurrent Processing , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[3]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[4]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[5]  Jerry M. Mendel,et al.  Tutorial on higher-order statistics (spectra) in signal processing and system theory: theoretical results and some applications , 1991, Proc. IEEE.

[6]  Benjamin Schrauwen,et al.  Recurrent Kernel Machines: Computing with Infinite Echo State Networks , 2012, Neural Computation.

[7]  K. Doya,et al.  Bifurcations in the learning of recurrent neural networks , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[8]  S. Wolfram.  Statistical mechanics of cellular automata , 1983 .

[9]  Melanie Mitchell,et al.  Computation in Cellular Automata: A Selected Review , 2005, Non-standard Computation.

[10]  Bernard De Baets,et al.  Phenomenological study of irregular cellular automata based on Lyapunov exponents and Jacobians. , 2010, Chaos.

[11]  A. Odlyzko,et al.  Algebraic properties of cellular automata , 1984 .

[12]  Stephen I. Gallant,et al.  Representing Objects, Relations, and Sequences , 2013, Neural Computation.

[13]  Ramón Alonso-Sanz,et al.  Elementary Cellular Automata with Elementary Memory Rules in Cells: The Case of Linear Rules , 2006, J. Cell. Autom..

[14]  Michel Verleysen,et al.  Flexible and Robust Bayesian Classification by Finite Mixture Models , 2004, ESANN.

[15]  José Manuel Ferrández,et al.  An efficient and expandable hardware implementation of multilayer cellular neural networks , 2013, Neurocomputing.

[16]  Yoav Goldberg,et al.  splitSVM: Fast, Space-Efficient, non-Heuristic, Polynomial Kernel Computation for NLP Applications , 2008, ACL.

[17]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[18]  Brian Kulis,et al.  Metric Learning: A Survey , 2013, Found. Trends Mach. Learn..

[19]  Riccardo Poli,et al.  Evolution of Cellular-automaton-based Associative Memories , 1998 .

[20]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[21]  Fatih Murat Porikli,et al.  Pedestrian Detection via Classification on Riemannian Manifolds , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Tony Jebara,et al.  Multi-task feature and kernel selection for SVMs , 2004, ICML.

[23]  Hui Cao,et al.  Approximate RBF Kernel SVM and Its Applications in Pedestrian Classification , 2008 .

[24]  Robert A. Legenstein,et al.  2007 Special Issue: Edge of chaos and prediction of computational performance for neural circuit models , 2007 .

[25]  Sepp Hochreiter,et al.  Untersuchungen zu dynamischen neuronalen Netzen , 1991 .

[26]  Santanu Chattopadhyay,et al.  Additive cellular automata : theory and applications , 1997 .

[27]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[28]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[29]  Anil K. Jain,et al.  A self-organizing network for hyperellipsoidal clustering (HEC) , 1996, IEEE Trans. Neural Networks.

[30]  Tony A. Plate,et al.  Holographic Reduced Representation: Distributed Representation for Cognitive Structures , 2003 .

[31]  Andrew Adamatzky,et al.  Computing in nonlinear media and automata collectives , 2001 .

[32]  Philip S. Yu,et al.  Finding generalized projected clusters in high dimensional spaces , 2000, SIGMOD '00.

[33]  Panagiotis Tzionas,et al.  A new, cellular automaton-based, nearest neighbor pattern classifier and its VLSI implementation , 1994, IEEE Trans. Very Large Scale Integr. Syst..

[34]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[35]  Niloy Ganguly,et al.  Theory of Additive Cellular Automata , 2007 .

[36]  Andrew Adamatzky,et al.  Identification of Cellular Automata , 2018, Encyclopedia of Complexity and Systems Science.

[37]  Thomas G. Dietterich.  Machine Learning for Sequential Data: A Review , 2002, SSPR/SPR.

[38]  Leon O. Chua,et al.  A Nonlinear Dynamics Perspective of Wolfram's New Kind of Science. Part X: Period-1 Rules , 2009, Int. J. Bifurc. Chaos.

[39]  Olga L. Bandman,et al.  Algebraic Properties of Cellular Automata: The Basis for Composition Technique , 2004, ACRI.

[40]  James P. Crutchfield,et al.  A Genetic Algorithm Discovers Particle-Based Computation in Cellular Automata , 1994, PPSN.

[41]  Genaro Juárez Martínez,et al.  Designing Complex Dynamics in Cellular Automata with Memory , 2013, Int. J. Bifurc. Chaos.

[42]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[43]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[44]  Nils Bertschinger,et al.  Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks , 2004, Neural Computation.

[45]  Yuichi Motai,et al.  Kernel Association for Classification and Prediction: A Survey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[46]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[47]  Herbert Jaeger,et al.  Long Short-Term Memory in Echo State Networks: Details of a Simulation Study , 2012 .

[48]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[49]  Olivier Debeir,et al.  Mixing Bagging and Multiple Feature Subsets to Improve Classification Accuracy of Decision Tree Combination , 2000 .

[50]  Fatih Murat Porikli,et al.  Region Covariance: A Fast Descriptor for Detection and Classification , 2006, ECCV.

[51]  Rolf Hoffmann,et al.  Implementing cellular automata in FPGA logic , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[52]  Marco Tomassini,et al.  Evolving Asynchronous and Scalable Non-uniform Cellular Automata , 1997, ICANNGA.

[53]  Johan A. K. Suykens,et al.  Fast Prediction with SVM Models Containing RBF Kernels , 2014, ArXiv.

[54]  Pentti Kanerva,et al.  Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors , 2009, Cognitive Computation.

[55]  Leon O. Chua,et al.  Cellular neural networks: applications , 1988 .

[56]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[57]  Genaro Juárez Martínez.  A Note on Elementary Cellular Automata Classification , 2013, J. Cell. Autom..

[58]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[59]  Matthew Cook,et al.  Universality in Elementary Cellular Automata , 2004, Complex Syst..

[60]  Hector Zenil,et al.  Asymptotic Behavior and ratios of Complexity in Cellular Automata , 2013, Int. J. Bifurc. Chaos.

[61]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[62]  Rong Jin,et al.  Distance Metric Learning: A Comprehensive Survey , 2006 .

[63]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .