Diverse sample generation with multi-branch conditional generative adversarial network for remote sensing objects detection

Abstract The remote sensing data is difficult to collect and lack of diversity, which extremely limits the performance of object detection on remote sensing images. In this paper, a multi-branch conditional generative adversarial network (MCGAN) is proposed to augment data for object detection in optical remote sensing images, which is the first GANs-based data augmentation framework proposed for this topic. We use MCGAN to generate the diverse objects based on the existing remote sensing datasets. The multi-branch dilated convolution and the classification branch are adopted into MCGAN to help the generator to generate the diverse and high-quality images. Meanwhile, an adaptive samples selection strategy based on the Faster R-CNN is proposed to select the samples for data augmentation from the objects generated by MCGAN, which can ensure the quality of new augmented training sets and improve the diversity of samples. Experiments based on NWPU VHR-10 and DOTA show that the objects generated by MCGAN have the higher quality compared with the objects generated by WGAN and LSGAN. And the mean average precision detected by the state-of-the-art object detection models used in the experiments has the satisfactory improvement after the MCGAN based data augmentation, which indicates that data augmentation by MCGAN can effectively improve the accuracy of remote sensing images object detection.

[1]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[2]  Rui Wang,et al.  Object instance detection with pruned Alexnet and extended training data , 2019, Signal Process. Image Commun..

[3]  Xuelong Li,et al.  Local structure learning in high resolution remote sensing image retrieval , 2016, Neurocomputing.

[4]  Hao Zhou,et al.  Faster R-CNN for marine organisms detection and recognition using data augmentation , 2019, Neurocomputing.

[5]  Han Dong,et al.  Building Damage Detection from Post-Event Aerial Imagery Using Single Shot Multibox Detector , 2019, Applied Sciences.

[6]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[7]  Derek Anderson,et al.  Fusion of an Ensemble of Augmented Image Detectors for Robust Object Detection , 2018, Sensors.

[8]  Nikos Komodakis,et al.  Cloud-Gan: Cloud Removal for Sentinel-2 Imagery Using a Cyclic Consistent Generative Adversarial Networks , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[9]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[10]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[11]  Xiaoyu Zhang,et al.  Bidirectional Active Learning: A Two-Way Exploration Into Unlabeled and Labeled Data Set , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Bing Liu,et al.  Siamese Convolutional Neural Networks for Remote Sensing Scene Classification , 2019, IEEE Geoscience and Remote Sensing Letters.

[13]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[15]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[16]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Yiannis S. Boutalis,et al.  Exploiting the generative adversarial framework for one-class multi-dimensional fault detection , 2019, Neurocomputing.

[18]  Ye Zhang,et al.  A light and faster regional convolutional neural network for object detection in optical remote sensing images , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[19]  Licheng Jiao,et al.  Classification of Hyperspectral Images Based on Multiclass Spatial–Spectral Generative Adversarial Networks , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Hichem Frigui,et al.  Robust fuzzy clustering for multiple instance regression , 2019, Pattern Recognit..

[21]  Xiaofeng He,et al.  Multiscale Object Detection in Infrared Streetscape Images Based on Deep Learning and Instance Level Data Augmentation , 2019 .

[22]  Yi Zheng,et al.  Remote sensing scene classification based on rotation-invariant feature learning and joint decision making , 2019, EURASIP J. Image Video Process..

[23]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[24]  Wujiao Dai,et al.  High Spatio-Temporal Resolution Deformation Time Series With the Fusion of InSAR and GNSS Data Using Spatio-Temporal Random Effect Model , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Zongxu Pan,et al.  Super-Resolution of Remote Sensing Images Based on Transferred Generative Adversarial Network , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[27]  Yi Li,et al.  R-FCN: Object Detection via Region-based Fully Convolutional Networks , 2016, NIPS.

[28]  Shunping Xiao,et al.  Deformable Faster R-CNN with Aggregating Multi-Layer Features for Partially Occluded Object Detection in Optical Remote Sensing Images , 2018, Remote. Sens..

[29]  Junwei Han,et al.  A Survey on Object Detection in Optical Remote Sensing Images , 2016, ArXiv.

[30]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[31]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[32]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  S. Shankar Sastry,et al.  Characterization and computation of local Nash equilibria in continuous games , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[34]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[35]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[36]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Zongjie Cao,et al.  Image Data Augmentation for SAR Sensor via Generative Adversarial Nets , 2019, IEEE Access.

[38]  Xiaodong Zhang,et al.  Geospatial Object Detection on High Resolution Remote Sensing Imagery Based on Double Multi-Scale Feature Pyramid Network , 2019, Remote. Sens..

[39]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Yandong Tang,et al.  Laplacian pyramid adversarial network for face completion , 2019, Pattern Recognit..

[41]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[42]  Yang Wang,et al.  Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation , 2016, ISVC.

[43]  Xinming Zhang,et al.  A novel framework for semantic segmentation with generative adversarial network , 2019, J. Vis. Commun. Image Represent..

[44]  Licheng Jiao,et al.  Multiobjective sparse ensemble learning by means of evolutionary algorithms , 2018, Decis. Support Syst..

[45]  Yu Kang,et al.  Deep spatiotemporal residual early-late fusion network for city region vehicle emission pollution prediction , 2019, Neurocomputing.

[46]  Jiebo Luo,et al.  DOTA: A Large-Scale Dataset for Object Detection in Aerial Images , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Yang Wang,et al.  Synthesizing remote sensing images by conditional adversarial networks , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[48]  Jun Zhou,et al.  Multiscale Visual Attention Networks for Object Detection in VHR Remote Sensing Images , 2019, IEEE Geoscience and Remote Sensing Letters.

[49]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[50]  Peng Li,et al.  Active semi-supervised learning based on self-expressive correlation with generative adversarial networks , 2019, Neurocomputing.

[51]  Junwei Han,et al.  Multi-class geospatial object detection and geographic image classification based on collection of part detectors , 2014 .