Parametric and nonparametric context models: A unified approach to scene parsing

Abstract In this paper a new nonparametric scene parsing approach is proposed which has three steps: image retrieval, label transferring and label gathering. In our approach, to incorporate the contextual knowledge in scene parsing, we propose to integrate both parametric and nonparametric context models into a unified framework. We adopt a co-occurrence graph to be our parametric context model to learn the co-occurrence frequency of objects. To consider different preferences of the co-occurring of one object with the other objects, the concept of co-occurring priority is introduced in this paper for the first time. Next, by using the learned co-occurrence graph and the context knowledge of the set of retrieved images, we propose new ways to incorporate contextual information in all three steps of nonparametric scene parsing approach. To evaluate our proposed approach, it is applied on MSRC-21 and SiftFlow datasets. The results show that our approach outperforms its competitors.

[1]  Antonio Torralba,et al.  Contextual Models for Object Detection Using Boosted Random Fields , 2004, NIPS.

[2]  Heesoo Myeong,et al.  Learning object relationships via graph-based context model , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Stephen Gould,et al.  Multiclass pixel labeling with non-local matching constraints , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[5]  Marcus Liwicki,et al.  Scene labeling with LSTM recurrent neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Lee Ryan,et al.  The effect of scene context on episodic object recognition: Parahippocampal cortex mediates memory encoding and retrieval success , 2007, Hippocampus.

[7]  Wei-Ping Zhu,et al.  Multi-scale context for scene labeling via flexible segmentation graph , 2016, Pattern Recognit..

[8]  Gang Wang,et al.  Integrating parametric and non-parametric models for scene labeling , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Jana Kosecka,et al.  Nonparametric Scene Parsing with Adaptive Feature Relevance and Semantic Context , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Lars Petersson,et al.  Sample and Filter: Nonparametric Scene Parsing via Efficient Filtering , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Gang Wang,et al.  DAG-Recurrent Neural Networks for Scene Labeling , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Junwei Han,et al.  Scene parsing using inference Embedded Deep Networks , 2016, Pattern Recognit..

[14]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  James J. Little,et al.  Scene parsing by nonparametric label transfer of content-adaptive windows , 2016, Comput. Vis. Image Underst..

[16]  Antonio Torralba,et al.  Contextual Priming for Object Detection , 2003, International Journal of Computer Vision.

[17]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[18]  Roberto Cipolla,et al.  Semantic texton forests for image categorization and segmentation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[20]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Kaia L. Vilberg,et al.  Brain Networks Underlying Episodic Memory Retrieval This Review Comes from a Themed Issue on Macrocircuits Memory Signals within the Mtl , 2022 .

[22]  Guosheng Lin,et al.  CRF Learning with CNN Features for Image Segmentation , 2015, Pattern Recognit..

[23]  Ronan Collobert,et al.  Recurrent Convolutional Neural Networks for Scene Labeling , 2014, ICML.

[24]  Shadrokh Samavi,et al.  A new fast approach to nonparametric scene parsing , 2014, Pattern Recognit. Lett..

[25]  Jürgen Schmidhuber,et al.  Multidimensional Recurrent Neural Networks , 2007 .

[26]  Svetlana Lazebnik,et al.  Superparsing , 2010, International Journal of Computer Vision.

[27]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Joost van de Weijer,et al.  Harmony Potentials , 2011, International Journal of Computer Vision.

[29]  Xuming He,et al.  Superpixel Graph Label Transfer with Learned Distance Metric , 2014, ECCV.

[30]  Svetlana Lazebnik,et al.  Understanding scenes on many levels , 2011, 2011 International Conference on Computer Vision.

[31]  Marios Savvides,et al.  Deep Contextual Recurrent Residual Networks for Scene Labeling , 2017, Pattern Recognit..

[32]  Cordelia Schmid,et al.  Coloring Local Feature Extraction , 2006, ECCV.

[33]  Stephen Gould,et al.  PatchMatchGraph: Building a Graph of Dense Patch Correspondences for Label Transfer , 2012, ECCV.

[34]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[35]  Andrea Vedaldi,et al.  Objects in Context , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[36]  Antonio Torralba,et al.  A Tree-Based Context Model for Object Recognition , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Rob Fergus,et al.  Nonparametric image parsing using adaptive neighbor sets , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  M. Bar.  Visual objects in context , 2004, Nature Reviews Neuroscience.

[39]  Sanja Fidler,et al.  Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[41]  David W. Jacobs,et al.  Deep hierarchical parsing for semantic segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Different Scenes , 2008, ECCV.

[43]  Sanja Fidler,et al.  The Role of Context for Object Detection and Semantic Segmentation in the Wild , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Antonio Torralba,et al.  Nonparametric Scene Parsing via Label Transfer , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.