Optimizing Echo State Networks for Static Pattern Recognition

Static pattern recognition requires a machine to classify an object on the basis of a combination of attributes and is typically performed using machine learning techniques such as support vector machines and multilayer perceptrons. Unusually, in this study, we applied a successful time-series processing neural network architecture, the echo state network (ESN), to a static pattern recognition task. The networks were presented with clamped input data patterns, but in this work, they were allowed to run until their output units delivered a stable set of output activations, in a similar fashion to previous work that focused on the behaviour of ESN reservoir units. Our aim was to see if the short-term memory developed by the reservoir and the clamped inputs could deliver improved overall classification accuracy. The study utilized a challenging, high dimensional, real-world plant species spectroradiometry classification dataset with the objective of accurately detecting one of the world’s top 100 invasive plant species. Surprisingly, the ESNs performed equally well with both unsettled and settled reservoirs. Delivering a classification accuracy of 96.60%, the clamped ESNs outperformed three widely used machine learning techniques, namely support vector machines, extreme learning machines and multilayer perceptrons. Contrary to past work, where inputs were clamped until reservoir stabilization, it was found that it was possible to obtain similar classification accuracy (96.49%) by clamping the input patterns for just two repeats. The chief contribution of this work is that a recurrent architecture can get good classification accuracy, even while the reservoir is still in an unstable state.

[1]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[2]  Jan de Leeuw,et al.  Discriminating species using hyperspectral indices at leaf and canopy scales. The International Arch , 2007 .

[3]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[4]  Douglas C. Montgomery,et al.  Introduction to Linear Regression Analysis, Solutions Manual (Wiley Series in Probability and Statistics) , 2007 .

[5]  Grgoire Montavon,et al.  Neural Networks: Tricks of the Trade , 2012, Lecture Notes in Computer Science.

[6]  C. Edwards,et al.  Characterising invasive non-native Rhododendron ponticum spectra signatures with spectroradiometry in the laboratory and field: Potential for remote mapping , 2013 .

[7]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[8]  Luís A. Alexandre,et al.  Reservoir computing for static pattern recognition , 2009, ESANN.

[9]  B. Matthews.  Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.

[10]  Benjamin Schrauwen,et al.  Defect Detection in Reinforced Concrete Using Random Neural Architectures , 2014, Comput. Aided Civ. Infrastructure Eng..

[11]  Abdelkader Benyettou,et al.  Novel Approach Using Echo State Networks for Microscopic Cellular Image Segmentation , 2016, Cognitive Computation.

[12]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[13]  Hendrik Van Brussel,et al.  Pruning and regularization in reservoir computing , 2009, Neurocomputing.

[14]  Luís A. Alexandre,et al.  Reservoir Size, Spectral Radius and Connectivity in Static Classification Problems , 2009, ICANN.

[15]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[16]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[17]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[18]  S. Murphy,et al.  The economic cost of invasive non-native species on Great Britain. Annexes 1 and 2 to the report. , 2010 .

[19]  Simone Scardapane,et al.  Semi-supervised Echo State Networks for Audio Classification , 2017, Cognitive Computation.

[20]  Mantas Lukosevicius,et al.  A Practical Guide to Applying Echo State Networks , 2012, Neural Networks: Tricks of the Trade.

[21]  Garrison W. Cottrell,et al.  2007 Special Issue: Learning grammatical structure with Echo State Networks , 2007 .

[22]  Jochen J. Steil,et al.  Attractor-based computation with reservoirs for online learning of inverse kinematics , 2009, ESANN.

[23]  Benjamin Schrauwen,et al.  Connectivity, Dynamics, and Memory in Reservoir Computing with Binary and Analog Neurons , 2010, Neural Computation.

[24]  Boris Bacic,et al.  Echo State Network for 3D Motion Pattern Indexing: A Case Study on Tennis Forehands , 2015, PSIVT.

[25]  Le Wang,et al.  INVASIVE SPECIES SPREAD MAPPING USING MULTI-RESOLUTION REMOTE SENSING DATA , 2008 .

[26]  J. Brian Gray,et al.  Introduction to Linear Regression Analysis , 2002, Technometrics.

[27]  M. Neteler,et al.  Benefits of hyperspectral remote sensing for tracking plant invasions , 2011 .

[28]  Henry Markram,et al.  The "Liquid Computer": A Novel Strategy for Real-Time Computing on Time Series , 2002 .

[29]  Petia D. Koprinkova-Hristova,et al.  Reservoir computing for emotion valence discrimination from EEG signals , 2017, Neurocomputing.

[30]  Herbert Jaeger,et al.  A tutorial on training recurrent neural networks , covering BPPT , RTRL , EKF and the " echo state network " approach - Semantic Scholar , 2005 .

[31]  Luís A. Alexandre,et al.  Benchmarking reservoir computing on time-independent classification tasks , 2009, 2009 International Joint Conference on Neural Networks.

[32]  Jochen J. Steil,et al.  Recurrence Enhances the Spatial Encoding of Static Inputs in Reservoir Networks , 2010, ICANN.

[33]  Masakazu Aono,et al.  A theoretical and experimental study of neuromorphic atomic switch networks for reservoir computing , 2013, Nanotechnology.