Photonic delay systems as machine learning implementations

Nonlinear photonic delay systems present interesting implementation platforms for machine learning models. They can be extremely fast, offer great degrees of parallelism and potentially consume far less power than digital processors. So far they have been successfully employed for signal processing using the Reservoir Computing paradigm. In this paper we show that their range of applicability can be greatly extended if we use gradient descent with backpropagation through time on a model of the system to optimize the input encoding of such systems. We perform physical experiments that demonstrate that the obtained input encodings work well in reality, and we show that optimized systems perform significantly better than the common Reservoir Computing approach. The results presented here demonstrate that common gradient descent techniques from machine learning may well be applicable on physical neuro-inspired analog computers.

[1]  Benjamin Schrauwen,et al.  Training and Analysing Deep Recurrent Neural Networks , 2013, NIPS.

[2]  Benjamin Schrauwen,et al.  Locomotion Without a Brain: Physical Reservoir Computing in Tensegrity Structures , 2013, Artificial Life.

[3]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Ilya Sutskever,et al.  Learning Recurrent Neural Networks with Hessian-Free Optimization , 2011, ICML.

[5]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[6]  Jan Danckaert,et al.  Constructing optimized binary masks for reservoir computing with delay systems , 2014, Scientific Reports.

[7]  Chrisantha Fernando,et al.  Pattern Recognition in a Bucket , 2003, ECAL.

[8]  Michiel Hermans,et al.  Optoelectronic Systems Trained With Backpropagation Through Time , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Carla Teixeira Lopes,et al.  TIMIT Acoustic-Phonetic Continuous Speech Corpus , 2012 .

[10]  Daniel Brunner,et al.  Parallel photonic information processing at gigabyte per second data rates using transient states , 2013, Nature Communications.

[11]  Benjamin Schrauwen,et al.  Optoelectronic Reservoir Computing , 2011, Scientific Reports.

[12]  Geert Morthier,et al.  Experimental demonstration of reservoir computing on a silicon photonics chip , 2014, Nature Communications.

[13]  Razvan Pascanu,et al.  Advances in optimizing recurrent networks , 2012, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[14]  L. Larger,et al.  Optoelectronic reservoir computing: tackling noise-induced performance degradation. , 2013, Optics express.

[15]  Jan Danckaert,et al.  Strongly asymmetric square waves in a time-delayed system. , 2012, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  J.J. Steil,et al.  Backpropagation-decorrelation: online recurrent learning with O(N) complexity , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[17]  Geoffrey E. Hinton,et al.  Generating Text with Recurrent Neural Networks , 2011, ICML.

[18]  Laurent Larger,et al.  Ikeda-based nonlinear delayed dynamics for application to secure optical transmission systems using chaos , 2004 .

[19]  Lawrence K. Saul,et al.  A fast online algorithm for large margin training of continuous density hidden Markov models , 2009, INTERSPEECH.

[20]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[21]  Tamir Hazan,et al.  PAC-Bayesian approach for minimization of phoneme error rate , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Benjamin Schrauwen,et al.  Toward optical signal processing using photonic reservoir computing. , 2008, Optics express.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  L Pesquera,et al.  Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. , 2012, Optics express.

[25]  Benjamin Schrauwen,et al.  Phoneme Recognition with Large Hierarchical Reservoirs , 2010, NIPS.

[26]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[27]  Benjamin Schrauwen,et al.  Automated Design of Complex Dynamic Systems , 2014, PloS one.

[28]  Yann LeCun,et al.  Discriminative Recurrent Sparse Auto-Encoders , 2013, ICLR.

[29]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[30]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[31]  Luca Maria Gambardella,et al.  Deep, Big, Simple Neural Nets for Handwritten Digit Recognition , 2010, Neural Computation.

[32]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[33]  Helmut Hauser,et al.  Towards a theoretical foundation for morphological computation with compliant bodies , 2011, Biological Cybernetics.