Gated networks: an inventory

Gated networks are networks that contain gating connections, in which the outputs of at least two neurons are multiplied. Initially, gated networks were used to learn relationships between two input sources, such as pixels from two images. More recently, they have been applied to learning activity recognition or multi-modal representations. The aims of this paper are threefold: 1) to explain the basic computations in gated networks to the non-expert, while adopting a standpoint that insists on their symmetric nature. 2) to serve as a quick reference guide to the recent literature, by providing an inventory of applications of these networks, as well as recent extensions to the basic architecture. 3) to suggest future research directions and applications.

[1]  Nitish Srivastava,et al.  Unsupervised Learning of Video Representations using LSTMs , 2015, ICML.

[2]  Bruno A. Olshausen,et al.  Principles of Image Representation in Visual Cortex , 2003 .

[3]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[4]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[5]  Jürgen Schmidhuber,et al.  LSTM can Solve Hard Long Time Lag Problems , 1996, NIPS.

[6]  Olivier Sigaud,et al.  Towards Deep Developmental Learning , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[7]  Roland Memisevic,et al.  Learning to Relate Images , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Daniel Jiwoong Im.  Analyzing the Dynamics of Gated Auto-encoders , 2014 .

[9]  Yann LeCun,et al.  Convolutional Learning of Spatio-temporal Features , 2010, ECCV.

[10]  Roland Memisevic,et al.  Modeling Deep Temporal Dependencies with Recurrent "Grammar Cells" , 2014, NIPS.

[11]  Manuel Lopes,et al.  Learning Object Affordances: From Sensory--Motor Coordination to Imitation , 2008, IEEE Transactions on Robotics.

[12]  Olivier Sigaud,et al.  Deep unsupervised network for multimodal perception, representation and classification , 2015, Robotics Auton. Syst..

[13]  Alain Droniou,et al.  Apprentissage de représentations et robotique développementale : quelques apports de l'apprentissage profond pour la robotique autonome. (Representation learning and developmental robotics : on the use of deep learning for autonomous robots) , 2015 .

[14]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[15]  Aapo Hyvärinen,et al.  Estimation of Non-Normalized Statistical Models by Score Matching , 2005, J. Mach. Learn. Res..

[16]  Geoffrey E. Hinton,et al.  Unsupervised Learning of Image Transformations , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Graham W. Taylor,et al.  "Mental Rotation" by Optimizing Transforming Distance , 2014, ArXiv.

[18]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[19]  Geoffrey E. Hinton,et al.  Gated Softmax Classification , 2010, NIPS.

[20]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[21]  Pascal Vincent,et al.  A Connection Between Score Matching and Denoising Autoencoders , 2011, Neural Computation.

[22]  Olivier Sigaud,et al.  Gated Autoencoders with Tied Input Weights , 2013, ICML.

[23]  Haitham Bou-Ammar,et al.  Factored four way conditional restricted Boltzmann machines for activity recognition , 2015, Pattern Recognit. Lett..

[24]  Afshin Dehghan,et al.  Who Do I Look Like? Determining Parent-Offspring Resemblance via Gated Autoencoders , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[26]  Roland Memisevic,et al.  On multi-view feature learning , 2012, ICML.

[27]  Oliver Brock,et al.  Patterns for Learning with Side Information , 2015, 1511.06429.

[28]  Roland Memisevic,et al.  Non-linear Latent Factor Models for Revealing Structure in High-dimensional Data , 2008 .

[29]  James Martens,et al.  Deep learning via Hessian-free optimization , 2010, ICML.

[30]  Geoffrey E. Hinton,et al.  Two Distributed-State Models For Generating High-Dimensional Time Series , 2011, J. Mach. Learn. Res..

[31]  Geoffrey E. Hinton,et al.  Generating Text with Recurrent Neural Networks , 2011, ICML.

[32]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[33]  Yoshua Bengio,et al.  Deep Learning of Representations: Looking Forward , 2013, SLSP.

[34]  Roland Memisevic,et al.  Learning Visual Odometry with a Convolutional Network , 2015, VISAPP.

[35]  Nando de Freitas,et al.  A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets , 2010, 2010 Information Theory and Applications Workshop (ITA).

[36]  Olivier Sigaud,et al.  Learning a repertoire of actions with deep neural networks , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[37]  Roland Memisevic,et al.  Modeling sequential data using higher-order relational features and predictive training , 2014, ArXiv.

[38]  Geoffrey E. Hinton,et al.  Factored conditional restricted Boltzmann Machines for modeling motion style , 2009, ICML '09.

[39]  Roland Memisevic,et al.  Gradient-based learning of higher-order image features , 2011, 2011 International Conference on Computer Vision.

[40]  Geoffrey E. Hinton,et al.  Learning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines , 2010, Neural Computation.

[41]  Graham W. Taylor,et al.  Generative Class-conditional Autoencoders , 2015, ICLR.

[42]  Droniou Alain,et al.  Gated Autoencoders with Tied Input Weights , 2013, ICML 2013.

[43]  IvaldiSerena,et al.  Deep unsupervised network for multimodal perception, representation and classification , 2015 .

[44]  Aapo Hyv.  Estimation of Non-Normalized Statistical Models by Score Matching , 2005 .

[45]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[46]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[47]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .