Scene parsing using inference Embedded Deep Networks

Effective features and graphical model are two key points for realizing high performance scene parsing. Recently, Convolutional Neural Networks (CNNs) have shown great ability of learning features and attained remarkable performance. However, most researches use CNNs and graphical model separately, and do not exploit full advantages of both methods. In order to achieve better performance, this work aims to design a novel neural network architecture called Inference Embedded Deep Networks (IEDNs), which incorporates a novel designed inference layer based on graphical model. Through the IEDNs, the network can learn hybrid features, the advantages of which are that they not only provide a powerful representation capturing hierarchical information, but also encapsulate spatial relationship information among adjacent objects. We apply the proposed networks to scene labeling, and several experiments are conducted on SIFT Flow and PASCAL VOC Dataset. The results demonstrate that the proposed IEDNs can achieve better performance. HighlightsWe design a novel structure of networks considering CRFs model as one type layer of deep neural networks.CRF is regarded as a layer of the network, therefore, the structural learning can be conducted explicitly.A novel feature encoding spatial relationship between objects in images is proposed.Feature fusing is adopted to learn intrinsic non-linear relationships between hierarchical and spatial features.