MaxFlow: a Convolutional Neural Network Based Optical Flow Algorithm for Large Displacement Estimation

Optical flow estimation is a basic problem in computer vision. FlowNet is the first convolutional neural network based optical algorithm that estimates optical flow by learning the relationship between image pair and the corresponding optical flow. In this paper, MaxFlow is proposed to improve the accuracy of FlowNet. The architecture of MaxFlow is similar to that of FlowNetSimple. MaxFlow uses two kinds of new layers, which are designed specially for estimating large displacements of small scale objects. The new down sampling layer makes the network to predict the maximum displacement in a region. Thus the large movements will not be missed. The new up sampling layer up samples the estimated optical flow fields without using any parameter. It simplifies the network without decreasing the accuracy of the network. Experiments on synthetic datasets and real datasets illustrate that the two new layers are effective and the accuracy of MaxFlow is higher than that of FlowNet.

[1]  Lior Wolf,et al.  PatchBatch: A Batch Augmented Loss for Optical Flow , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Andreas Geiger,et al.  Deep Discrete Flow , 2016, ACCV.

[3]  Vladlen Koltun,et al.  Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Didier Stricker,et al.  Supplementary material of : CNN-based Patch Matching for Optical Flow with Thresholded Hinge Embedding Loss , 2017 .

[6]  Christian Heipke,et al.  Discrete Optimization for Optical Flow , 2015, GCPR.

[7]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[8]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[9]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[10]  Yunsong Li,et al.  Efficient Coarse-to-Fine Patch Match for Large Displacement Optical Flow , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Lior Wolf,et al.  InterpoNet, a Brain Inspired Neural Network for Optical Flow Dense Interpolation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Didier Stricker,et al.  Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[15]  Andrea Vedaldi,et al.  Fully-trainable deep matching , 2016, BMVC.

[16]  Lior Wolf,et al.  Optical Flow Requires Multiple Strategies (but Only One Network) , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Jia Xu,et al.  Accurate Optical Flow via Direct Cost Volume Processing , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Stefano Soatto,et al.  S2F: Slow-to-Fast Interpolator Flow , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[20]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).