Decision Fusion With Multiple Spatial Supports by Conditional Random Fields

Classification of remotely sensed images into land cover or land use is highly dependent on geographical information at least at two levels. First, land cover classes are observed in a spatially smooth domain separated by sharp region boundaries. Second, land classes and observation scale are also tightly intertwined: they tend to be consistent within areas of homogeneous appearance, or regions, in the sense that all pixels within a roof should be classified as roof, independently on the spatial support used for the classification. In this paper, we follow these two observations and encode them as priors in an energy minimization framework based on conditional random fields (CRFs), where classification results obtained at pixel and region levels are probabilistically fused. The aim is to enforce the final maps to be consistent not only in their own spatial supports (pixel and region) but also across supports, i.e., by getting the predictions on the pixel lattice and on the set of regions to agree. To this end, we define an energy function with three terms: 1) a data term for the individual elements in each support (support-specific nodes); 2) spatial regularization terms in a neighborhood for each of the supports (support-specific edges); and 3) a regularization term between individual pixels and the region containing each of them (intersupports edges). We utilize these priors in a unified energy minimization problem that can be optimized by standard solvers. The proposed 2L↯CRF model consists of a CRF defined over a bipartite graph, i.e., two interconnected layers within a single graph accounting for interlattice connections. 2L↯CRF is tested on two very high-resolution data sets involving submetric satellite and subdecimeter aerial data. In all cases, 2L↯CRF improves the result obtained by the independent base model (either random forests or convolutional neural networks) and by standard CRF models enforcing smoothness in the spatial domain.

[1]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Gabriele Moser,et al.  Weight Parameter Optimization by the Ho–Kashyap Algorithm in MRF Models for Supervised Image Classification , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Nicolas Courty,et al.  Multiclass feature learning for hyperspectral image classification: sparse and hierarchical solutions , 2015, ArXiv.

[4]  Konrad Schindler,et al.  An Overview and Comparison of Smooth Labeling Methods for Land-Cover Classification , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[5]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[6]  A. Willsky.  Multiresolution Markov models for signal and image processing , 2002, Proc. IEEE.

[7]  Stan Z. Li,et al.  Markov Random Field Modeling in Image Analysis , 2001, Computer Science Workbench.

[8]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Andrew McCallum,et al.  An Introduction to Conditional Random Fields , 2010, Found. Trends Mach. Learn..

[10]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Pablo J. Zarco-Tejada,et al.  Thermal and Narrowband Multispectral Remote Sensing for Vegetation Monitoring From an Unmanned Aerial Vehicle , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Michele Volpi,et al.  Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[13]  Pietro Perona,et al.  Cataloging Public Objects Using Aerial and Street-Level Images — Urban Trees , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Alain Rakotomamonjy,et al.  Automatic Feature Learning for Spatio-Spectral Image Classification With Sparse SVM , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Gabriele Moser,et al.  Multiresolution Supervised Classification of Panchromatic and Multispectral Images by Markov Random Fields and Graph Cuts , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Silvana G. Dellepiane,et al.  Synthetic aperture radar image segmentation by a detail preserving Markov random field approach , 1997, IEEE Trans. Geosci. Remote. Sens..

[19]  Gabriele Moser,et al.  Getting pixels and regions to agree with conditional random fields , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[20]  Xiao Xiang Zhu,et al.  Deep learning in remote sensing: a review , 2017, ArXiv.

[21]  J. Besag.  On the Statistical Analysis of Dirty Pictures , 1986 .

[22]  Jean Ponce,et al.  Discriminative clustering for image co-segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Anil K. Jain,et al.  A Markov random field model for classification of multisource satellite imagery , 1996, IEEE Trans. Geosci. Remote. Sens..

[25]  Chao Luo,et al.  SCENE-LAYOUT COMPATIBLE CONDITIONAL RANDOM FIELD FOR CLASSIFYING TERRESTRIAL LASER POINT CLOUDS , 2014 .

[26]  William J. Emery,et al.  A neural network approach using multi-scale textural metrics from very high-resolution panchromatic imagery for urban land-use classification , 2009 .

[27]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[28]  Michele Volpi,et al.  Structured prediction for urban scene semantic segmentation with geographic context , 2015, 2015 Joint Urban Remote Sensing Event (JURSE).

[29]  Michael S. Brown,et al.  A framework for reducing ink-bleed in old documents , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Christian Heipke,et al.  A higher order conditional random field model for simultaneous classification of land cover and land use , 2017 .

[31]  Li Hua,et al.  Detection of Damaged Rooftop Areas From High-Resolution Aerial Images Based on Visual Bag-of-Words Model , 2016, IEEE Geoscience and Remote Sensing Letters.

[32]  Darius Burschka,et al.  Toward a Fully Autonomous UAV: Research Platform for Indoor and Outdoor Urban Search and Rescue , 2012, IEEE Robotics & Automation Magazine.

[33]  Josiane Zerubia,et al.  Hierarchical Multiple Markov Chain Model for Unsupervised Texture Segmentation , 2009, IEEE Transactions on Image Processing.

[34]  Pascal Fua,et al.  Conditional Random Fields for multi-camera object detection , 2011, 2011 International Conference on Computer Vision.

[35]  Michele Volpi,et al.  Detecting animals in African Savanna with UAVs and the crowds , 2017, ArXiv.

[36]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[37]  Rama Chellappa,et al.  Entropy rate superpixel segmentation , 2011, CVPR 2011.

[38]  Xuezhi Feng,et al.  Cosegmentation for Object-Based Building Change Detection From High-Resolution Remotely Sensed Images , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[39]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[40]  Pierre Alliez,et al.  Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Christian Heipke,et al.  Conditional Random Fields for Multitemporal and Multiscale Classification of Optical Satellite Imagery , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Sebastiano B. Serpico,et al.  A Markov random field approach to spatio-temporal contextual image classification , 2003, IEEE Trans. Geosci. Remote. Sens..

[43]  Pierre Alliez,et al.  High-Resolution Semantic Labeling with Convolutional Neural Networks , 2016 .

[44]  Jitendra Malik,et al.  Hypercolumns for object segmentation and fine-grained localization , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Chih-Jen Lin,et al.  Probability Estimates for Multi-class Classification by Pairwise Coupling , 2003, J. Mach. Learn. Res..

[46]  P. H. Swain,et al.  Bayesian classification in a time-varying environment , 1978 .

[47]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[48]  Yang Xiao,et al.  Efficient Airport Detection Using Line Segment Detector and Fisher Vector Representation , 2016, IEEE Geoscience and Remote Sensing Letters.

[49]  Jamie Sherrah,et al.  Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery , 2016, ArXiv.

[50]  Alexandre Boulch,et al.  Benchmarking classification of earth-observation data: From learning explicit features to convolutional networks , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[51]  Gui-Song Xia,et al.  Bag-of-Visual-Words Scene Classifier With Local and Global Features for High Spatial Resolution Remote Sensing Imagery , 2016, IEEE Geoscience and Remote Sensing Letters.

[52]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[53]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[54]  Konrad Schindler,et al.  Road networks as collections of minimum cost paths , 2015 .

[55]  Andrew Blake,et al.  Cosegmentation of Image Pairs by Histogram Matching - Incorporating a Global Constraint into MRFs , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[56]  Nikolaos Grammalidis,et al.  Building Detection Using Enhanced HOG–LBP Features and Region Refinement Processes , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[57]  Pushmeet Kohli,et al.  Robust Higher Order Potentials for Enforcing Label Consistency , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Uwe Stilla,et al.  Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection , 2016, ISPRS Journal of Photogrammetry and Remote Sensing.

[59]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[60]  Alexandre Boulch,et al.  Processing of Extremely High-Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest–Part A: 2-D Contest , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[61]  Devis Tuia,et al.  Geospatial Correspondences for Multimodal Registration , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  智一 吉田,et al.  Efficient Graph-Based Image Segmentationを用いた圃場図自動作成手法の検討 , 2014 .

[63]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[64]  Gabriele Moser,et al.  A New Cascade Model for the Hierarchical Joint Classification of Multitemporal and Multiresolution Remote Sensing Data , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[65]  Thomas Blaschke,et al.  Object based image analysis for remote sensing , 2010 .

[66]  Erle C. Ellis,et al.  Using lightweight unmanned aerial vehicles to monitor tropical forest recovery , 2015 .

[67]  Michele Volpi,et al.  Semantic segmentation of urban scenes by learning local class interactions , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[68]  Pierre Alliez,et al.  High-Resolution Aerial Image Labeling With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[69]  Qian Du,et al.  Remote Sensing Image Scene Classification Using Multi-Scale Completed Local Binary Patterns and Fisher Vectors , 2016, Remote. Sens..

[70]  Nikos Komodakis,et al.  Rotation Equivariant Vector Field Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[71]  Alejandro C. Frery,et al.  When Data Do Not Bring Information: A Case Study in Markov Random Fields Estimation , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[72]  Bertrand Le Saux,et al.  Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks , 2016, ACCV.

[73]  Josiane Zerubia,et al.  Markov Random Fields in Image Segmentation , 2012, Found. Trends Signal Process..

[74]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields , 2006, ECCV.

[75]  Jon Atli Benediktsson,et al.  Morphological Attribute Profiles for the Analysis of Very High Resolution Images , 2010, IEEE Transactions on Geoscience and Remote Sensing.