Unorganized Machines: From Turing's Ideas to Modern Connectionist Approaches

This work presents a discussion about the relationship between the contributions of Alan Turing – the centenary of whose birth is celebrated in 2012 – to the field of artificial neural networks and modern unorganized machines: reservoir computing (RC) approaches and extreme learning machines (ELMs). Firstly, the authors review Turing’s connectionist proposals and also expose the fundamentals of the main RC paradigms – echo state networks and liquid state machines, - as well as of the design and training of ELMs. Throughout this exposition, the main points of contact between Turing’s ideas and these modern perspectives are outlined, being, then, duly summarized in the second and final part of the work. This paper is useful in offering a distinct appreciation of Turing’s pioneering contributions to the field of neural networks and also in indicating some perspectives for the future development of the field that may arise from the synergy between these views.

[1]  Minoru Asada,et al.  Initialization and self‐organized optimization of recurrent neural network connectivity , 2009, HFSP journal.

[2]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[3]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[4]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[5]  Hans-Paul Schwefel,et al.  Numerical Optimization of Computer Models , 1982 .

[6]  Eleonora Bilotta,et al.  Cellular Automata and Complex Systems: Methods for Modeling Biological Phenomena , 2010 .

[7]  Andrew Hodges,et al.  Alan Turing: The Enigma , 1983 .

[8]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[9]  B. J. Copeland,et al.  Alan Turing’s Forgotten Ideas in Computer Science , 1999 .

[10]  Wolfgang Maass,et al.  Liquid Computing , 2007, CiE.

[11]  David Leavitt.  The Man Who Knew Too Much: Alan Turing and the Invention of the Computer , 2005 .

[12]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[13]  Narasimhan Sundararajan,et al.  Fully complex extreme learning machine , 2005, Neurocomputing.

[14]  D. M. Hutton,et al.  The Essential Turing , 2007 .

[15]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[16]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[17]  Hava T. Siegelmann,et al.  The Dynamic Universality of Sigmoidal Neural Networks , 1996, Inf. Comput..

[18]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[19]  Filip De Turck,et al.  A novel time series analysis approach for prediction of dialysis in critically ill patients using echo-state networks , 2010, BMC Medical Informatics Decis. Mak..

[20]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[21]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[22]  Christof Teuscher.  Turing's connectionism - an investigation of neural network architectures , 2002, Discrete mathematics and theoretical computer science.

[23]  A. Turing.  On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[24]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[25]  M. A. O'Neil,et al.  The connectional organization of the cortico-thalamic system of the cat. , 1999, Cerebral cortex.

[26]  Benjamin Schrauwen,et al.  Extending reservoir computing with random static projections: a hybrid between extreme learning and RC , 2010, ESANN.

[27]  José Carlos Príncipe,et al.  Water Inflow Forecasting using the Echo State Network: a Brazilian Case Study , 2007, 2007 International Joint Conference on Neural Networks.

[28]  B. Jack Copeland,et al.  On Alan Turing's anticipation of connectionism , 1996, Synthese.

[29]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[30]  Fernando José Von Zuben,et al.  An extended echo state network using Volterra filtering and principal component analysis , 2012, Neural Networks.

[31]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[32]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[33]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[34]  Fernando José Von Zuben,et al.  An echo state network architecture based on volterra filtering and PCA with application to the channel equalization problem , 2011, The 2011 International Joint Conference on Neural Networks.

[35]  Chee Kheong Siew,et al.  Incremental extreme learning machine with fully complex hidden nodes , 2008, Neurocomputing.

[36]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[37]  Frederico G. Guimarães,et al.  Memetic and Evolutionary Design of Wireless Sensor Networks Based on Complex Network Characteristics , 2010, Int. J. Nat. Comput. Res..

[38]  Stephen McGlinchey,et al.  Biologically Inspired Artificial Intelligence for Computer Games , 2007 .

[39]  José Carlos Príncipe,et al.  Analysis and Design of Echo State Networks , 2007, Neural Computation.

[40]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[41]  Tomás Lang,et al.  Introduction to Digital Systems , 1998 .

[42]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[43]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[44]  Arie Yeredor,et al.  ICA over finite fields - Separability and algorithms , 2012, Signal Process..

[45]  Henry Markram,et al.  Fading memory and kernel properties of generic cortical microcircuit models , 2004, Journal of Physiology-Paris.

[46]  Benoît Frénay,et al.  Parameter-insensitive kernel in extreme learning for non-linear support vector regression , 2011, Neurocomputing.

[47]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[48]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[49]  Christof Teuscher,et al.  A Revival of Turing's Forgotten Connectionist Ideas: Exploring Unorganized Machines , 2000, NCPW.

[50]  Tadashi Yamazaki,et al.  The cerebellum as a liquid state machine , 2007, Neural Networks.

[51]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[52]  Eleonora Bilotta,et al.  Cellular Automata Metrics , 2010 .

[53]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[54]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.