Joint Depth and Semantic Inference from a Single Image via Elastic Conditional Random Field

The estimations of depth and regional semantics from a single image have traditionally been considered as two separated problems. In this paper, we argue that these two tasks provide complementary information, which therefore can be performed jointly to reinforce individual tasks in terms of both accuracy and speed. In particular, we propose an Elastic Conditional Random Field (E-CRF) deployed upon superpixel segmentations, which models the interdependency between depth and semantics to refine each other in an iterative manner. Differing from the traditional CRFs, E-CRF makes edges elastically hidden/emergent during inference to conduct fast Loopy Belief Propagation, while explicitly modeling the depth-label interdependency to achieve high inference accuracy. Moreover, the Structured Support Vector Machine (SSVM) is further introduced to drastically speed up the inference. We have conducted extensive evaluations on both Make3D and NYU benchmark datasets, which demonstrated that our E-CRF method significantly outperforms state-of-the-art techniques in terms of precision, while significantly accelerating the inference speed (2-3 orders of magnitude). HighlightsEfficient joint inference of depth estimation and region labeling from a single image.Our first contribution is an efficient generative model called Elastic Conditional Random Field (E-CRF) to capture the interdependency between depth and labeling, along with the spatial dependency among neighborhood superpixels.Our second contribution is to further accelerate the above LBP-based generative inference from a large-margin perspective by using a Structured Support Vector Machine (SSVM).