Modeling the visual world: reconstruction and neural episodic representation

Modeling the visual world is the cornerstone of solving many important problems in engineering and science. Those problems include augmented/virtual reality (AR/VR), artificial intelligence (AI) and robotics. The goal of modeling is usually to distill the perceived world into a form which is suitable for solving a range of tasks. Over the years of research, multiple approaches to modeling the world were proposed. Currently, three most widely used approaches are: explicitly reconstructing the world from images, keeping the knowledge implicitly as weights of a neural network and representing the world as a collection of neural embedding instances. The first approach relies on human geometric priors about the process by which the images were obtained. Those priors provide geometric constraints which allow to determine the 3D model which is consistent with the observed evidence. The second and the third approaches are based on learning — that is, fitting the weights of a parametric function while solving some human-designed optimization problem. The second approach does not keep explicit memory of individual instances of experience — all the information is distilled into the weights. The third approach preserves the instances of interaction with the world as individual embedding instances. Embedding neural network extracts those instances from experience and other neural networks operate on them — for example, to determine similarities between pairs of experiences. In this thesis, we analyze different modeling approaches and make three novel contributions to them. First, we propose a very accurate reconstruction method called Ray Po-