Reconstructing perceived faces from brain activations with deep adversarial neural decoding

Here, we present a novel approach to solve the problem of reconstructing perceived stimuli from brain responses by combining probabilistic inference with deep learning. Our approach first inverts the linear transformation from latent features to brain responses with maximum a posteriori estimation and then inverts the nonlinear transformation from perceived stimuli to latent features with adversarial training of convolutional neural networks. We test our approach with a functional magnetic resonance imaging experiment and show that it can generate state-of-the-art reconstructions of perceived faces from brain activations.

[1]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[2]  P. Matthew Bronstad,et al.  Skin and Bones: The Contribution of Skin Tone and Facial Structure to Racial Prototypicality Ratings , 2012, PloS one.

[3]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[4]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[5]  Michael Eickenberg,et al.  Seeing it all: Convolutional network layers map the function of the human visual system , 2017, NeuroImage.

[6]  Tomoyasu Horikawa,et al.  Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features , 2016, Front. Comput. Neurosci..

[7]  D. Perrett,et al.  Men's strategic preferences for femininity in female faces. , 2014, British journal of psychology.

[8]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[9]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[10]  Mark W. Woolrich,et al.  FSL , 2012, NeuroImage.

[11]  Russell A. Poldrack,et al.  Deconvolving BOLD activation in event-related designs for multivoxel pattern classification analyses , 2012, NeuroImage.

[12]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[13]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[14]  Kaare Brandt Petersen,et al.  The Matrix Cookbook , 2006 .

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[17]  Marcel A. J. van Gerven Unsupervised Learning of Features for Bayesian Decoding in Functional Magnetic Resonance Imaging , 2013 .

[18]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[19]  Brice A. Kuhl,et al.  Reconstructing Perceived and Retrieved Faces from Activity Patterns in Lateral Parietal Cortex , 2016, The Journal of Neuroscience.

[20]  Marcel A. J. van Gerven,et al.  A primer on encoding models in sensory neuroscience , 2017 .

[21]  Brice A. Kuhl,et al.  Neural portraits of perception: Reconstructing face images from evoked brain activity , 2014, NeuroImage.

[22]  Tomoyasu Horikawa,et al.  Generic decoding of seen and imagined objects using hierarchical visual features , 2015, Nature Communications.

[23]  Antonio Torralba,et al.  Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence , 2016, Scientific Reports.

[24]  H. P. Op de Beeck,et al.  Representations of Facial Identity Information in the Ventral Visual Stream Investigated with Multivoxel Pattern Analyses , 2013, The Journal of Neuroscience.

[25]  M. A. Goodale,et al.  What is the best fixation target? The effect of target shape on stability of fixational eye movements , 2013, Vision Research.

[26]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[27]  Tom Heskes,et al.  Neural Decoding with Hierarchical Generative Models , 2010, Neural Computation.

[28]  Marcel van Gerven,et al.  Increasingly complex representations of natural movies across the dorsal stream are shared between subjects , 2017, NeuroImage.

[29]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[30]  D. C. Donderi,et al.  Compressed file length predicts search time and errors on visual displays , 2005, Displays.

[31]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[32]  Changde Du,et al.  Sharing deep generative representation for perceived image reconstruction from human brain activity , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[33]  Kenta Oono,et al.  Chainer : a Next-Generation Open Source Framework for Deep Learning , 2015 .

[34]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[35]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[36]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[37]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[38]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[39]  David I. Perrett,et al.  The role of sexually dimorphic skin colour and shape in attractiveness of male faces , 2016 .

[40]  Joshua Correll,et al.  The Chicago face database: A free stimulus set of faces and norming data , 2015, Behavior research methods.

[41]  Tom Heskes,et al.  A Linear Gaussian Framework for Decoding of Perceived Images , 2012, 2012 Second International Workshop on Pattern Recognition in NeuroImaging.

[42]  Kurt Gray,et al.  The MR2: A multi-racial, mega-resolution database of facial stimuli , 2016, Behavior research methods.

[43]  David I. Perrett,et al.  Neural and behavioral responses to attractiveness in adult and infant faces , 2014, Neuroscience & Biobehavioral Reviews.

[44]  Tom Heskes,et al.  Gaussian mixture models and semantic gating improve reconstructions from human brain activity , 2015, Front. Comput. Neurosci..

[45]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Richard H. A. H. Jacobs,et al.  Liking versus Complexity: Decomposing the Inverted U-curve , 2016, Front. Hum. Neurosci..

[47]  Marcel A. J. van Gerven,et al.  Brains on Beats , 2016, NIPS.

[48]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[49]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Nikolaus Kriegeskorte,et al.  Deep neural networks: a new framework for modelling biological vision and brain information processing , 2015, bioRxiv.

[51]  Andrea Bergmann,et al.  Statistical Parametric Mapping The Analysis Of Functional Brain Images , 2016 .

[52]  Marcel van Gerven,et al.  Convolutional Sketch Inversion , 2016, ECCV Workshops.

[53]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[54]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[55]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[56]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[57]  D. Perrett,et al.  Cross-cultural perception of trustworthiness: The effect of ethnicity features on evaluation of faces’ observed trustworthiness across four samples , 2014 .

[58]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[59]  Tom Heskes,et al.  Linear reconstruction of perceived images from human brain activity , 2013, NeuroImage.

[60]  Jean-Baptiste Poline,et al.  Inverse retinotopy: Inferring the visual content of images from brain activation patterns , 2006, NeuroImage.

[61]  Marcel van Gerven,et al.  Modeling the Dynamics of Human Brain Activity with Recurrent Neural Networks , 2016, Front. Comput. Neurosci..

[62]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[63]  D. Perrett,et al.  Facial shape and judgements of female attractiveness , 1994, Nature.

[64]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.