Attentive Sequential Neural Processes

Sequential Neural Processes (SNP) is a new class of models that temporally extends Neural Processes (NP) and can meta-learn a sequence of stochastic processes. This learned function however under-fits the provided contexts as is also the case in NP. Applying attention to the contexts resolves this in NP but simply extending this to SNP and applying attention on a buffer of context history is sub-optimal as our findings show. In this paper, we propose Attentive Sequential Neural Processes (ASNP) which resolves the under-fitting in SNP by introducing a novel imaginary context, modeled as a latent variable, over which the attention can then be applied. We evaluate our model on 1D Gaussian Processes regression and 2D moving MNIST/CelebA regression. We apply ASNP to implement Attentive Temporal-GQN and further evaluate on the moving CelebA task.

[1]  Demis Hassabis,et al.  Neural Episodic Control , 2017, ICML.

[2]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[3]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[4]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[5]  Sungjin Ahn,et al.  Sequential Neural Processes , 2019, NeurIPS.

[6]  Yee Whye Teh,et al.  Conditional Neural Processes , 2018, ICML.

[7]  Murray Shanahan,et al.  Consistent Generative Query Networks , 2018, ArXiv.

[8]  Max Welling,et al.  VAE with a VampPrior , 2017, AISTATS.

[9]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[10]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[11]  Daniel L. Schacter,et al.  Constructive memory: past and future , 2012, Dialogues in clinical neuroscience.

[12]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[13]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[14]  Koray Kavukcuoglu,et al.  Neural scene representation and rendering , 2018, Science.

[15]  H. Eichenbaum.  Memory: Organization and Control. , 2017, Annual review of psychology.

[16]  Yee Whye Teh,et al.  Set Transformer , 2018, ICML.

[17]  Daan Wierstra,et al.  Towards Conceptual Compression , 2016, NIPS.

[18]  Klamer Schutte,et al.  The Functional Neural Process , 2019, NeurIPS.

[19]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[20]  Yee Whye Teh,et al.  Attentive Neural Processes , 2019, ICLR.

[21]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Fabio Viola,et al.  Learning models for visual 3D localization with implicit mapping , 2018, ArXiv.

[23]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.