Branch Feature Fusion Convolution Network for Remote Sensing Scene Classification

Convolutional neural networks (CNNs) have outstanding advantages in the classification of remote sensing scenes. Deep CNN models with better classification performance typically have high complexity, whereas shallow CNN models with low complexity rarely achieve good classification performance for remote sensing images with complex spatial structures. In this article, we proposed a new lightweight CNN classification method based on branch feature fusion (LCNN-BFF) for remote sensing scene classification. In contrast to a conventional single linear convolution structure, the proposed model had a bilinear feature extraction structure. The BFF method was utilized to fuse the feature information extracted from the two branches, which improved the classification accuracy. In addition, combining depthwise separable convolution and conventional convolution to extract image features greatly reduced the complexity of the model on the premise of ensuring the accuracy of classification. We tested the method on four standard datasets. The experimental results showed that, compared with recent classification methods, the number of weight parameters of the proposed method only accounted for less than 5% of the other methods; however, the classification accuracy was equivalent to or even superior to certain high-performance classification methods.

[1]  Qianqing Qin,et al.  Scene Classification Based on Multiscale Convolutional Neural Network , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[3]  Yishu Liu,et al.  Scene Classification Based on Two-Stage Deep Feature Fusion , 2018, IEEE Geoscience and Remote Sensing Letters.

[4]  Shutao Li,et al.  Remote Sensing Scene Classification Using Multilayer Stacked Covariance Pooling , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[5]  Honglak Lee,et al.  Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units , 2016, ICML.

[6]  Antonio Plaza,et al.  Scale-Free Convolutional Neural Network for Remote Sensing Scene Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Min Wang,et al.  Very high resolution remote sensing image classification with SEEDS-CNN and scale effect analysis for superpixel CNN classification , 2018, International Journal of Remote Sensing.

[8]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[9]  Bing Liu,et al.  Siamese Convolutional Neural Networks for Remote Sensing Scene Classification , 2019, IEEE Geoscience and Remote Sensing Letters.

[10]  Sim Heng Ong,et al.  Multifeature energy optimization framework and parameter adjustment-based nonrigid point set registration , 2018, Journal of Applied Remote Sensing.

[11]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[12]  Xiaoqiang Lu,et al.  A Coarse-to-Fine Semi-Supervised Change Detection for Multispectral Images , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[13]  Nan Li,et al.  Positional Context Aggregation Network for Remote Sensing Scene Classification , 2020, IEEE Geoscience and Remote Sensing Letters.

[14]  Cong Lin,et al.  Integrating Multilayer Features of Convolutional Neural Networks for Remote Sensing Scene Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[16]  Xiaoqiang Lu,et al.  Remote Sensing Image Scene Classification: Benchmark and State of the Art , 2017, Proceedings of the IEEE.

[17]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[18]  Hongxun Yao,et al.  Deep Feature Fusion for VHR Remote Sensing Scene Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[19]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[20]  Xiangtao Zheng,et al.  Multisource Compensation Network for Remote Sensing Cross-Domain Scene Classification , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Lei Guo,et al.  Object Detection in Optical Remote Sensing Images Based on Weakly Supervised Learning and High-Level Feature Learning , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Liwei Wang,et al.  The Expressive Power of Neural Networks: A View from the Width , 2017, NIPS.

[23]  Gong Cheng,et al.  P-CNN: Part-Based Convolutional Neural Networks for Fine-Grained Visual Categorization , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Xuelong Li,et al.  Bidirectional Adaptive Feature Fusion for Remote Sensing Scene Classification , 2017, CCCV.

[26]  Fei Hu,et al.  Semi-Supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks , 2020, IEEE Access.

[27]  Xiangtao Zheng,et al.  Remote Sensing Scene Classification by Unsupervised Representation Learning , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Lei Guo,et al.  When Deep Learning Meets Metric Learning: Remote Sensing Image Scene Classification via Learning Discriminative CNNs , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Junwei Han,et al.  A Survey on Object Detection in Optical Remote Sensing Images , 2016, ArXiv.

[30]  Ran Cao,et al.  Self-Attention-Based Deep Feature Fusion for Remote Sensing Scene Classification , 2021, IEEE Geoscience and Remote Sensing Letters.

[31]  Xiangtao Zheng,et al.  Exploring Models and Data for Remote Sensing Image Caption Generation , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Ping Zhong,et al.  Diversity-Promoting Deep Structural Metric Learning for Remote Sensing Scene Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[33]  Anil M. Cheriyadat,et al.  Unsupervised Feature Learning for Aerial Scene Classification , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[34]  Xiangtao Zheng,et al.  Remote Sensing Scene Classification by Gated Bidirectional Network , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[35]  Jefersson Alex dos Santos,et al.  Towards better exploiting convolutional neural networks for remote sensing scene classification , 2016, Pattern Recognit..

[36]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[37]  Lei Guo,et al.  Auto-encoder-based shared mid-level visual dictionary learning for scene classification using very high resolution remote sensing images , 2015, IET Comput. Vis..

[38]  Lin Chen,et al.  A New Image Recognition and Classification Method Combining Transfer Learning Algorithm and MobileNet Model for Welding Defects , 2020, IEEE Access.

[39]  Qing Liu,et al.  Accurate Object Localization in Remote Sensing Images Based on Convolutional Neural Networks , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[40]  Fuxian Liu,et al.  Convolutional neural network based heterogeneous transfer learning for remote-sensing scene classification , 2019, International Journal of Remote Sensing.

[41]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[42]  Xie Bangquan,et al.  Real-Time Embedded Traffic Sign Recognition Using Efficient Convolutional Neural Network , 2019, IEEE Access.

[43]  Shiming Xiang,et al.  Aggregating Rich Hierarchical Features for Scene Classification in Remote Sensing Imagery , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[44]  Ke Li,et al.  Rotation-Insensitive and Context-Augmented Object Detection in Remote Sensing Images , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[45]  Lei Guo,et al.  Exploring Hierarchical Convolutional Features for Hyperspectral Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Shiyue Zhang,et al.  Aggregated Deep Fisher Feature for VHR Remote Sensing Scene Classification , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[47]  Zhou Yang,et al.  A novel two-stage scene classification model based on feature variable significance in high-resolution remote sensing , 2020 .

[48]  Hao Sun,et al.  A Feature Aggregation Convolutional Neural Network for Remote Sensing Scene Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[49]  Yanjiang Wang,et al.  Weighted Spatial Pyramid Matching Collaborative Representation for Remote-Sensing-Image Scene Classification , 2019, Remote. Sens..

[50]  Lijun Zhao,et al.  Remote Sensing Image Scene Classification Using CNN-CapsNet , 2019, Remote. Sens..

[51]  Chen Wang,et al.  Multiple resolution block feature for remote-sensing scene classification , 2019, International Journal of Remote Sensing.

[52]  Antonio Plaza,et al.  Skip-Connected Covariance Network for Remote Sensing Scene Classification , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[53]  Tong Zhang,et al.  Deep Learning Based Feature Selection for Remote Sensing Scene Classification , 2015, IEEE Geoscience and Remote Sensing Letters.

[54]  Ruyi Feng,et al.  An Improved Pretraining Strategy-Based Scene Classification With Deep Learning , 2020, IEEE Geoscience and Remote Sensing Letters.

[55]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[56]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[57]  Yang Yang,et al.  Image registration using two-layer cascade reciprocal pipeline and context-aware dissimilarity measure , 2020, Neurocomputing.

[58]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[59]  Gui-Song Xia,et al.  Bag-of-Visual-Words Scene Classifier With Local and Global Features for High Spatial Resolution Remote Sensing Imagery , 2016, IEEE Geoscience and Remote Sensing Letters.

[60]  Gui-Song Xia,et al.  AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[61]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[62]  Yongjun Zhang,et al.  A Lightweight and Discriminative Model for Remote Sensing Scene Classification With Multidilation Pooling Module , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[63]  Yi Zheng,et al.  Remote sensing scene classification based on rotation-invariant feature learning and joint decision making , 2019, EURASIP J. Image Video Process..

[64]  Xiaoqiang Lu,et al.  Learning deep event models for crowd anomaly detection , 2017, Neurocomputing.

[65]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[66]  Mohamed Farah,et al.  Remote Sensing Scene Classification Using Convolutional Features and Deep Forest Classifier , 2019, IEEE Geoscience and Remote Sensing Letters.

[67]  Antonio J. Plaza,et al.  A New Spatial–Spectral Feature Extraction Method for Hyperspectral Images Using Local Covariance Matrix Representation , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[68]  Hassan Ghassemian,et al.  A review of remote sensing image fusion methods , 2016, Inf. Fusion.

[69]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[70]  Bo Du,et al.  Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art , 2016, IEEE Geoscience and Remote Sensing Magazine.

[71]  Shutao Li,et al.  Hyperspectral Image Classification With Deep Feature Fusion Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[72]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[73]  Xueming Qian,et al.  Semantic Annotation of High-Resolution Satellite Images via Weakly Supervised Learning , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[74]  Shawn D. Newsam,et al.  Bag-of-visual-words and spatial extensions for land-use classification , 2010, GIS '10.

[75]  Pierre Alliez,et al.  High-Resolution Aerial Image Labeling With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.