Automatic generation of dense non-rigid optical flow

There hardly exists any large-scale datasets with dense optical flow of non-rigid motion from real-world imagery as of today. The reason lies mainly in the difficulty of human annotation to generate optical flow ground-truth. To circumvent the need for human annotation, we propose a framework to automatically generate optical flow from real-world videos. The method extracts and matches objects from video frames to compute initial constraints, and applies a deformation over the objects of interest to obtain dense optical flow fields. We propose several ways to augment the optical flow variations. Extensive experimental results show that training on our automatically generated optical flow outperforms methods that are trained on rigid synthetic data using FlowNet-S, PWC-Net, and LiteFlowNet. Datasets and algorithms of our optical flow generation framework is available at https://github.com/lhoangan/arap_flow.