Synthetic training data for deep neural networks on visual correspondence tasks

In the realm of deep learning for computer vision tasks, the best performing models tend to be trained with supervision, i.e. with a training dataset that contains ground-truth annotations which the model is expected to match. Visual tasks are particularly interesting because humans rely mostly on their eyes for almost everything they do; we attribute great importance to our visual perception of the world, and we have developed methods to produce visually realistic simulations of this world for purposes of entertainment, communication, and research. These same methods enable the creation of synthetic training data: rendered views of virtual worlds with annotations that are more extensive and accurate than anything a human could label with justifiable time and effort. In this thesis, we motivate and describe the making of large synthetic datasets for low-level correspondence matching problems. We used these datasets to train deep neural networks for the fundamental vision tasks of optical flow and stereo disparity estimation, achieving a new state of the art at the time of their publication. We further isolate individual design components that make up an optical flow dataset, and analyze their contributions to the data’s suitability for training. Finally, we use our results to create new datasets for specific real-world scenarios, thereby demonstrating that data engineering is a viable and practicable method for improving the performance of neural networks. Complementary to optimizations that operate on a network itself such as those of architecture, loss function or model capacity, data is a design dimension that can be varied even if the learning algorithm is a black box.