Active learning for interactive satellite image change detection

We introduce in this paper a novel active learning algorithm for satellite image change detection. The proposed solution is interactive and based on a question & answer model, which asks an oracle (annotator) the most informative questions about the relevance of sampled satellite image pairs, and according to the oracle’s responses, updates a decision function iteratively. We investigate a novel framework which models the probability that samples are relevant; this probability is obtained by minimizing an objective function capturing representativity, diversity and ambiguity. Only data with a high probability according to these criteria are selected and displayed to the oracle for further annotation. Extensive experiments on the task of satellite image change detection after natural hazards (namely tornadoes) show the relevance of the proposed method against the related work. keywords: Frugal learning, interactive satellite image change detection.

[1]  Francis R. Bach,et al.  Active learning for misspecified generalized linear models , 2006, NIPS.

[2]  Laurens van der Maaten,et al.  Self-Supervised Learning of Pretext-Invariant Representations , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Quentin Oliveau,et al.  Learning Attribute Representations for Remote Sensing Ship Category Classification , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[4]  Hichem Sahbi,et al.  Directed Acyclic Graph Kernels for Action Recognition , 2013, 2013 IEEE International Conference on Computer Vision.

[5]  Hichem Sahbi,et al.  Robust matching by dynamic space warping for accurate face recognition , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[6]  Nikolaos Papanikolopoulos,et al.  Multi-class active learning for image classification , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Marco Diani,et al.  Introductory view of anomalous change detection in hyperspectral images within a theoretical gaussian framework , 2017, IEEE Aerospace and Electronic Systems Magazine.

[8]  Yutaka Matsuo,et al.  Epipolar-Guided Deep Object Matching for Scene Change Detection , 2020, ArXiv.

[9]  Y. Zhong,et al.  Hi-UCD: A Large-scale Dataset for Urban Semantic Change Detection in Remote Sensing Imagery , 2020, ArXiv.

[10]  Hichem Sahbi,et al.  High Order Stochastic Graphlet Embedding for Graph-Based Pattern Recognition , 2017, ArXiv.

[11]  Geoffrey E. Hinton,et al.  Zero-shot Learning with Semantic Output Codes , 2009, NIPS.

[12]  David Cohn,et al.  Active Learning , 2010, Encyclopedia of Machine Learning.

[13]  Xi Peng,et al.  A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Shuhui Bu,et al.  Change detection in images using shape-aware siamese convolutional network , 2020, Eng. Appl. Artif. Intell..

[15]  Hichem Sahbi,et al.  Nonlinear Deep Kernel Learning for Image Annotation , 2017, IEEE Transactions on Image Processing.

[16]  Hichem Sahbi,et al.  Deep kernel map networks for image annotation , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]  Jian Wu,et al.  An Active Learning Approach with Uncertainty, Representativeness, and Diversity , 2014, TheScientificWorldJournal.

[18]  Hao Chen,et al.  A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection , 2020, Remote. Sens..

[19]  Paolo Favaro,et al.  Boosting Self-Supervised Learning via Knowledge Transfer , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Sergey Levine,et al.  Time-Contrastive Networks: Self-Supervised Learning from Video , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[21]  Shaogang Gong,et al.  Semantic Autoencoder for Zero-Shot Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Hichem Sahbi,et al.  Semi supervised deep kernel design for image annotation , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Hichem Sahbi,et al.  Face detection using coarse-to-fine support vector classifiers , 2002, Proceedings. International Conference on Image Processing.

[24]  Hichem Sahbi.  ImageCLEF annotation with explicit context-aware kernel maps , 2015, International Journal of Multimedia Information Retrieval.

[25]  Jie Tang,et al.  Self-Supervised Learning: Generative or Contrastive , 2020, IEEE Transactions on Knowledge and Data Engineering.

[26]  Hichem Sahbi,et al.  Camera pose estimation using Visual Servoing for aerial video change detection , 2012, 2012 IEEE International Geoscience and Remote Sensing Symposium.

[27]  Zhi-Hua Zhou,et al.  A brief introduction to weakly supervised learning , 2018 .

[28]  Hichem Sahbi,et al.  Spatio-temporal interaction for aerial video change detection , 2012, 2012 IEEE International Geoscience and Remote Sensing Symposium.

[29]  Xiaobo Jin,et al.  Attentive Region Embedding Network for Zero-Shot Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Mohammed Bennamoun,et al.  Weakly Supervised Change Detection in a Pair of Images , 2016, ArXiv.

[31]  Hichem Sahbi,et al.  Mid-level features and spatio-temporal context for activity recognition , 2012, Pattern Recognit..

[32]  Hichem Sahbi,et al.  A particular Gaussian mixture model for clustering and its application to image retrieval , 2008, Soft Comput..

[33]  Gilles Aubert,et al.  A contrast equalization procedure for change detection algorithms: Applications to remotely sensed images of urban areas , 2008, 2008 19th International Conference on Pattern Recognition.

[34]  A. Doulamis,et al.  Building Change Detection using Semantic Segmentation on Analogue Aerial Photos , 2018 .

[35]  Ihab Sbeity,et al.  Building Change Detection in Aerial Images , 2019, BDCSIntell.

[36]  Liangpei Zhang,et al.  A Deeply Supervised Attention Metric-Based Network and an Open Aerial Image Dataset for Remote Sensing Change Detection , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Bo Du,et al.  Slow Feature Analysis for Change Detection in Multispectral Imagery , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Hichem Sahbi,et al.  Laplacian deep kernel learning for image annotation , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[39]  Hichem Sahbi,et al.  Using entropy for image and video authentication watermarks , 2006, Electronic Imaging.

[40]  Ted Briscoe,et al.  Weakly Supervised Learning for Hedge Classification in Scientific Literature , 2007, ACL.

[41]  Xueming Qian,et al.  Semantic Annotation of High-Resolution Satellite Images via Weakly Supervised Learning , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Weilin Huang,et al.  CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images , 2018, ECCV.

[43]  Jinsong Deng,et al.  PCA‐based land‐use change detection and analysis using multitemporal and multisensor satellite data , 2008 .

[44]  Hichem Sahbi,et al.  Constrained optical flow for aerial image change detection , 2011, 2011 IEEE International Geoscience and Remote Sensing Symposium.

[45]  Hichem Sahbi,et al.  Transductive Kernel Map Learning and Its Application Image Annotation , 2012, BMVC.

[46]  Hichem Sahbi.  Misalignment resilient CCA for interactive satellite image change detection , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[47]  Zoubin Ghahramani,et al.  Bayesian Active Learning for Classification and Preference Learning , 2011, ArXiv.

[48]  Hichem Sahbi,et al.  Relevance feedback for satellite image change detection , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[49]  Donghai Xie,et al.  S2Looking: A Satellite Side-Looking Dataset for Building Change Detection , 2021, Remote. Sens..

[50]  F. Fleuret,et al.  Scale-Invariance of Support Vector Machines based on the Triangular Kernel , 2001 .

[51]  Mi Zhang,et al.  Object-level change detection with a dual correlation attention-guided detector , 2021, ISPRS Journal of Photogrammetry and Remote Sensing.

[52]  Cunbao Ma,et al.  Aerial image change detection using dual regions of interest networks , 2019, Neurocomputing.

[53]  Kevin Gimpel,et al.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , 2019, ICLR.

[54]  Geoffrey J. Hay,et al.  Object-based change detection , 2012 .

[55]  Hichem Sahbi,et al.  Kernel PCA for similarity invariant shape recognition , 2007, Neurocomputing.

[56]  Wei-Lun Chao,et al.  Synthesized Classifiers for Zero-Shot Learning , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Sanjoy Dasgupta,et al.  Analysis of a greedy active learning strategy , 2004, NIPS.

[58]  Hichem Sahbi,et al.  Interactive Satellite Image Change Detection With Context-Aware Canonical Correlation Analysis , 2017, IEEE Geoscience and Remote Sensing Letters.

[59]  Hichem Sahbi,et al.  Manifold learning using robust Graph Laplacian for interactive image search , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[60]  Piyush Rai,et al.  Generalized Zero-Shot Learning via Synthesized Examples , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  Qing Li,et al.  Mask-CDNet: A mask based pixel change detection network , 2020, Neurocomputing.

[62]  Radu Timofte,et al.  A Weakly Supervised Convolutional Network for Change Segmentation and Classification , 2020, ACCV Workshops.

[63]  Yu. V. Vizilter,et al.  CHANGE DETECTION VIA MORPHOLOGICAL COMPARATIVE FILTERS , 2016 .

[64]  Wenzhong Shi,et al.  Change Detection Based on Artificial Intelligence: State-of-the-Art and Challenges , 2020, Remote. Sens..

[65]  Q. Guo,et al.  Reducing Mis-registration and Shadow Effects on Change Detection in Wetlands , 2011 .

[66]  Evgeny Burnaev,et al.  Targeted change detection in remote sensing images , 2019, International Conference on Machine Vision.

[67]  Zhi-Hua Zhou,et al.  Towards Safe Weakly Supervised Learning , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  John R. Jensen,et al.  Object‐based change detection using correlation image analysis and image segmentation , 2008 .

[69]  Hichem Sahbi,et al.  Deep representation design from deep kernel networks , 2019, Pattern Recognit..

[70]  Andreas Savakis,et al.  Unsupervised change detection using Spatial Transformer Networks , 2016, 2016 IEEE Western New York Image and Signal Processing Workshop (WNYISPW).

[71]  Badrinath Roysam,et al.  Image change detection algorithms: a systematic survey , 2005, IEEE Transactions on Image Processing.

[72]  Hichem Sahbi,et al.  Context-Dependent Kernels for Object Classification , 2011, IEEE Trans. Pattern Anal. Mach. Intell..

[73]  Trevor Campbell,et al.  Automated Scalable Bayesian Inference via Hilbert Coresets , 2017, J. Mach. Learn. Res..

[74]  Alexander Kolesnikov,et al.  Revisiting Self-Supervised Visual Representation Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Sébastien Leprince,et al.  Automatic and Precise Orthorectification, Coregistration, and Subpixel Correlation of Satellite Images, Application to Ground Deformation Measurements , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[76]  Venkatesh Saligrama,et al.  Zero-Shot Learning via Semantic Similarity Embedding , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[77]  Patrick C. Hytla.  Multi-Ratio Fusion Change Detection Framework with Adaptive Statistical Thresholding , 2016 .

[78]  King-Sun Fu,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Publication Information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Hichem Sahbi,et al.  Bags-of-daglets for action recognition , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[80]  Oriol Vinyals,et al.  Matching Networks for One Shot Learning , 2016, NIPS.

[81]  Hichem Sahbi,et al.  Coarse-to-Fine Deep Kernel Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[82]  Tao Xiang,et al.  Learning a Deep Embedding Model for Zero-Shot Learning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Hichem Sahbi,et al.  From coarse to fine skin and face detection , 2000, ACM Multimedia.

[84]  Jihwan P. Choi,et al.  Local Similarity Siamese Network for Urban Land Change Detection on Remote Sensing Images , 2021, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[85]  Lorenzo Bruzzone,et al.  Earthquake Damage Assessment of Buildings Using VHR Optical and SAR Imagery , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[86]  Luke S. Zettlemoyer,et al.  Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions , 2013, TACL.

[87]  Ersin Yumer,et al.  Self-supervised Learning of Motion Capture , 2017, NIPS.

[88]  Francesca Bovolo,et al.  Sequential Spectral Change Vector Analysis for Iteratively Discovering and Detecting Multiple Changes in Hyperspectral Images , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[89]  Hichem Sahbi,et al.  Multi-view object matching and tracking using canonical correlation analysis , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[90]  Alexander Kolesnikov,et al.  S4L: Self-Supervised Semi-Supervised Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[91]  Zoubin Ghahramani,et al.  Deep Bayesian Active Learning with Image Data , 2017, ICML.

[92]  D. Konstantinidis.  Building detection for monitoring of urban changes , 2017 .

[93]  Chunyan Miao,et al.  A Survey of Zero-Shot Learning , 2019, ACM Trans. Intell. Syst. Technol..

[94]  Dawn Song,et al.  Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty , 2019, NeurIPS.

[95]  Hichem Sahbi,et al.  Nonlinear Cross-View Sample Enrichment for Action Recognition , 2014, ECCV Workshops.

[96]  Daniel P. Huttenlocher,et al.  Weakly Supervised Learning of Part-Based Spatial Models for Visual Object Recognition , 2006, ECCV.

[97]  Mohammed Bennamoun,et al.  Learning deep structured network for weakly supervised change detection , 2016, IJCAI.

[98]  Masashi Matsuoka,et al.  A Method for Detecting Buildings Destroyed by the 2011 Tohoku Earthquake and Tsunami Using Multitemporal TerraSAR-X Data , 2015, IEEE Geoscience and Remote Sensing Letters.

[99]  Thomas B. Pollard,et al.  Comprehensive 3-d change detection using volumetric appearance modeling , 2009 .

[100]  Hichem Sahbi,et al.  A Hierarchy of Support Vector Machines for Pattern Detection , 2006, J. Mach. Learn. Res..

[101]  Hichem Sahbi,et al.  Applying interest operators in semi-fragile video watermarking , 2005, IS&T/SPIE Electronic Imaging.

[102]  Mats I. Pettersson,et al.  Change detection in aerial images using a Kendall's TAU distance pattern correlation , 2016, 2016 6th European Workshop on Visual Information Processing (EUVIP).

[103]  Alberto Del Bimbo,et al.  Context-Dependent Logo Matching and Recognition , 2013, IEEE Transactions on Image Processing.

[104]  Hichem Sahbi,et al.  Kernel methods and scale invariance using the triangular kernel , 2004 .

[105]  Allan Aasbjerg Nielsen,et al.  The Regularized Iteratively Reweighted MAD Method for Change Detection in Multi- and Hyperspectral Data , 2007, IEEE Transactions on Image Processing.

[106]  Evgeny Burnaev,et al.  Procedural Synthesis of Remote Sensing Images for Robust Change Detection with Neural Networks , 2019, ISNN.

[107]  Nabil Zerrouki,et al.  Statistical Monitoring of Changes to Land Cover , 2018, IEEE Geoscience and Remote Sensing Letters.

[108]  Soma Biswas,et al.  Preserving Semantic Relations for Zero-Shot Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[109]  Mark J. Carlotto,et al.  Detecting change in images with parallax , 2007, SPIE Defense + Commercial Sensing.

[110]  Yang Wu,et al.  Meta-Learning Transferable Active Learning Policies by Deep Reinforcement Learning , 2018, ArXiv.

[111]  Hichem Sahbi,et al.  CNRS - TELECOM ParisTech at ImageCLEF 2013 Scalable Concept Image Annotation Task: Winning Annotations with Context Dependent SVMs , 2013, CLEF.

[112]  Douglas G. Macharet,et al.  Fully Convolutional Siamese Autoencoder for Change Detection in UAV Aerial Images , 2020, IEEE Geoscience and Remote Sensing Letters.