Operational Calculus on Programming Spaces

In this work we develop an algebraic language that represents a formal calculus for deep learning and is, at the same time a model which enables implementations and investigations of programs. To this purpose, we develop an abstract computational model of automatically differentiable programs. In the model, programs are elements of op. cit. programming spaces. Programs are viewed as maps from a finite-dimensional vector space to itself op. cit. virtual memory space. Virtual memory space is an algebra of programs, an algebraic data structure (one can calculate with). The elements of the virtual memory space give the expansion of a program into an infinite tensor series. We define a differential operator on programming spaces and, using its powers, implement the general shift operator and the operator of program composition. The algebraic language constructed in this way is a complete model of deep learning. It enables us to express programs in such a way, that their properties may be derived from their source codes.

[1]  Ziga Sajovic,et al.  Implementation of Operational calculus on programming spaces with Applications , 2016, ArXiv.

[2]  V. Sajovic.  Implementing Operational calculus on programming spaces for Differentiable computing , 2016, 1612.02731.

[3]  Sergio Gomez Colmenarejo,et al.  Hybrid computing using a neural network with dynamic external memory , 2016, Nature.

[4]  Jack J. Dongarra,et al.  High-Performance Tensor Contractions for GPUs , 2016, ICCS.

[5]  John Salvatier,et al.  Theano: A Python framework for fast computation of mathematical expressions , 2016, ArXiv.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Nando de Freitas,et al.  Neural Programmer-Interpreters , 2015, ICLR.

[8]  Paul I. Barton,et al.  A vector forward mode of automatic differentiation for generalized derivative evaluation , 2015, Optim. Methods Softw..

[9]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[10]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[11]  Robin J. Hogan,et al.  Fast Reverse-Mode Automatic Differentiation using Expression Templates in C++ , 2014, ACM Trans. Math. Softw..

[12]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[13]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[14]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[15]  Andrew Y. Ng,et al.  Parsing Natural Scenes and Natural Language with Recursive Neural Networks , 2011, ICML.

[16]  M. Girolami,et al.  Riemann manifold Langevin and Hamiltonian Monte Carlo methods , 2011, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[17]  Yoshua Bengio,et al.  Gradient-Based Optimization of Hyperparameters , 2000, Neural Computation.

[18]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[19]  R. Feynman.  An Operator calculus having applications in quantum electrodynamics , 1951 .

[20]  H. T. H. PIAGGIO,et al.  The Operational Calculus , 1943, Nature.

[21]  John R. Carson,et al.  The Heaviside operational calculus , 1922 .

[22]  Alexander Mordvintsev,et al.  Inceptionism: Going Deeper into Neural Networks , 2015 .

[23]  Barak A. Pearlmutter,et al.  Putting the Automatic Back into AD:Part II, Dynamic, Automatic, Nestable, and Fast(CVS: 1.1) , 2008 .

[24]  Barak A. Pearlmutter,et al.  Putting the Automatic Back into AD:Part I, What’s Wrong (CVS: 1.1) , 2008 .

[25]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[26]  W. Rindler,et al.  Gravitation and Spacetime , 1977 .