InterpoNet, a Brain Inspired Neural Network for Optical Flow Dense Interpolation

Sparse-to-dense interpolation for optical flow is a fundamental phase in the pipeline of most of the leading optical flow estimation algorithms. The current state-of-the-art method for interpolation, EpicFlow, is a local average method based on an edge aware geodesic distance. We propose a new data-driven sparse-to-dense interpolation algorithm based on a fully convolutional network. We draw inspiration from the filling-in process in the visual cortex and introduce lateral dependencies between neurons and multi-layer supervision into our learning process. We also show the importance of the image contour to the learning process. Our method is robust and outperforms EpicFlow on competitive optical flow benchmarks with several underlying matching algorithms. This leads to state-of-the-art performance on the Sintel and KITTI 2012 benchmarks.

[1]  Didier Stricker,et al.  Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Hamutal Slovin,et al.  A Contrast and Surface Code Explains Complex Responses to Black and White Stimuli in V1 , 2014, The Journal of Neuroscience.

[3]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[4]  Ken Nakayama,et al.  Brightness perception and filling-in , 1991, Vision Research.

[5]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  A. Dale,et al.  The Representation of Illusory and Real Contours in Human Cortical Visual Areas Revealed by Functional Magnetic Resonance Imaging , 1999, The Journal of Neuroscience.

[7]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Michael J. Black,et al.  Optical Flow with Semantic Segmentation and Localized Layers , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Frank Tong,et al.  Filling-in of visual phantoms in the human brain , 2005, Nature Neuroscience.

[10]  Bernhard Schölkopf,et al.  Mask-Specific Inpainting with Deep Neural Networks , 2014, GCPR.

[11]  Hong Zhou,et al.  The coding of uniform colour figures in monkey visual cortex , 2003, The Journal of physiology.

[12]  Hong Zhou,et al.  Searching For The Neural Mechanism Of Color Filling-In , 2003 .

[13]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[14]  C. Lawrence Zitnick,et al.  Structured Forests for Fast Edge Detection , 2013, 2013 IEEE International Conference on Computer Vision.

[15]  C. Hung,et al.  Cortical processing of a brightness illusion. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[17]  P. Kellman,et al.  A common mechanism for illusory and occluded object completion. , 1998, Journal of experimental psychology. Human perception and performance.

[18]  Takeo Watanabe,et al.  The primary visual cortex fills in color , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[19]  B. Pinna,et al.  Surface color from boundaries: a new ‘watercolor’ illusion , 2001, Vision Research.

[20]  Peter De Weerd,et al.  Responses of cells in monkey visual cortex during perceptual filling-in of an artificial scotoma , 1995, Nature.

[21]  Hamutal Slovin,et al.  Representation of Color Surfaces in V1: Edge Enhancement and Unfilled Holes , 2015, The Journal of Neuroscience.

[22]  Michael J. Black,et al.  A Naturalistic Open Source Movie for Optical Flow Evaluation , 2012, ECCV.

[23]  R. von der Heydt,et al.  Color Filling-in under Steady Fixation: Behavioral Demonstration in Monkeys and Humans , 1999, Perception.

[24]  Christian Heipke,et al.  Discrete Optimization for Optical Flow , 2015, GCPR.

[25]  H. Neumann,et al.  The Role of Attention in Figure-Ground Segregation in Areas V1 and V4 of the Visual Cortex , 2012, Neuron.

[26]  Pushmeet Kohli,et al.  The Global Patch Collider , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  R. L. Gregory,et al.  Perceptual filling in of artificially induced scotomas in human vision , 1991, Nature.

[28]  Yi Wang,et al.  Representation of surface luminance and contrast in primary visual cortex. , 2012, Cerebral cortex.

[29]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  H. Komatsu.  The neural mechanisms of perceptual filling-in , 2006, Nature Reviews Neuroscience.

[31]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  M. Paradiso,et al.  V1 response timing and surface filling-in. , 2008, Journal of neurophysiology.

[33]  H. Komatsu,et al.  Neural Responses in the Retinotopic Representation of the Blind Spot in the Macaque V1 to Stimuli for Perceptual Filling-In , 2000, The Journal of Neuroscience.

[34]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[35]  Andreas Geiger,et al.  Deep Discrete Flow , 2016, ACCV.

[36]  Xiaolin Hu,et al.  Recurrent convolutional neural network for object recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Vladlen Koltun,et al.  Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Michael S. Brown,et al.  SPM-BP: Sped-Up PatchMatch Belief Propagation for Continuous MRFs , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[39]  T. Sejnowski,et al.  Representation of Color Stimuli in Awake Macaque Primary Visual Cortex , 2003, Neuron.

[40]  Yunsong Li,et al.  Efficient Coarse-to-Fine Patch Match for Large Displacement Optical Flow , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  V S Ramachandran,et al.  Blind Spots , 2011, The American journal of bioethics : AJOB.

[42]  Stefan Roth,et al.  Joint Optical Flow and Temporally Consistent Semantic Segmentation , 2016, ECCV Workshops.