Effectively modeling piecewise planar urban scenes based on structure priors and CNN

Dear editor, Piecewise planar stereo methods can approximately reconstruct the complete structures of a scene by overcoming challenging difficulties (e.g., poorly textured regions) that pixel-level stereos appear powerless. In general, these methods have three basic steps: (1) over-segmenting the image into several regions (superpixels) without overlapping; (2) generating candidate planes from initial 3D points; (3) assigning the optimal plane for each superpixel using a global method. However, such methods can be unreliable and inefficient because of three reasons: (1) inaccurate image oversegmentation (generating superpixels based only on low-level image features); (2) incomplete candidate planes (failing to generate complete candidate planes from sparse 3D points); (3) unreliable regularization terms (forcing two neighboring superpixels with similar appearances to be assigned the same plane). To solve these problems, in this study, a novel plane assignment cost is first constructed by incorporating structure priors and high-level image features obtained by Convolutional Neural Network (CNN). Then, the scene structures are reconstructed in a progressive manner that jointly optimizes image regions and their associated planes, followed by a global plane assignment optimization under a Markov random field (MRF) framework. Methodology. Given the current image Ir and its left and right neighboring images {Ni} (i = 1, 2), we define the following cost of assigning a plane Hs to a superpixel s ∈ Ir.