Scene Reconstruction Using MRF Optimization with Image Content Adaptive Energy Functions

Multi-view scene reconstruction from multiple uncalibrated images can be solved by two stages of processing: first, a sparse reconstruction using Structure From Motion (SFM), and second, a surface reconstruction using optimization of Markov random field (MRF). This paper focuses on the second step, assuming that a set of sparse feature points have been reconstructed and the cameras have been calibrated by SFM. The multi-view surface reconstruction is formulated as an image-based multi-labeling problem solved using MRF optimization via graph cut. First, we construct a 2D triangular mesh on the reference image, based on the image segmentation results provided by an existing segmentation process. By doing this, we expect that each triangle in the mesh is well aligned with the object boundaries, and a minimum number of triangles are generated to represent the 3D surface. Second, various objective and heuristic depth cues such as the slanting cue, are combined to define the local penalty and interaction energies. Third, these local energies are adapted to the local image content, based on the results from some simple content analysis techniques. The experimental results show that the proposed method is able to well the preserve the depth discontinuity because of the image content adaptive local energies.