Multi-Evidence and Multi-Modal Fusion Network for Ground-Based Cloud Recognition

In recent times, deep neural networks have drawn much attention in ground-based cloud recognition. Yet such kind of approaches simply center upon learning global features from visual information, which causes incomplete representations for ground-based clouds. In this paper, we propose a novel method named multi-evidence and multi-modal fusion network (MMFN) for ground-based cloud recognition, which could learn extended cloud information by fusing heterogeneous features in a unified framework. Namely, MMFN exploits multiple pieces of evidence, i.e., global and local visual features, from ground-based cloud images using the main network and the attentive network. In the attentive network, local visual features are extracted from attentive maps which are obtained by refining salient patterns from convolutional activation maps. Meanwhile, the multi-modal network in MMFN learns multi-modal features for ground-based cloud. To fully fuse the multi-modal and multi-evidence visual features, we design two fusion layers in MMFN to incorporate multi-modal features with global and local visual features, respectively. Furthermore, we release the first multi-modal ground-based cloud dataset named MGCD which not only contains the ground-based cloud images but also contains the multi-modal information corresponding to each cloud image. The MMFN is evaluated on MGCD and achieves a classification accuracy of 88.63% comparative to the state-of-the-art methods, which validates its effectiveness for ground-based cloud recognition.

[1]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[3]  Yunxue Shao,et al.  Salient local binary pattern for ground-based cloud classification , 2013, Acta Meteorologica Sinica.

[4]  Zhenhua Guo,et al.  A Completed Modeling of Local Binary Pattern Operator for Texture Classification , 2010, IEEE Transactions on Image Processing.

[5]  Mei Li,et al.  Deep tensor fusion network for multimodal ground-based cloud classification in weather station networks , 2020, Ad Hoc Networks.

[6]  Young-Koo Lee,et al.  Feature Fusion of Deep Spatial Features and Handcrafted Spatiotemporal Features for Human Action Recognition , 2019, Sensors.

[7]  Mei Li,et al.  Dual Guided Loss for Ground-Based Cloud Classification in Weather Station Networks , 2019, IEEE Access.

[8]  Tao Mei,et al.  Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Apichat Heednacram,et al.  FFT features and hierarchical classification algorithms for cloud images , 2018, Eng. Appl. Artif. Intell..

[10]  Apichat Heednacram,et al.  Feature extraction techniques for ground-based cloud type classification , 2015, Expert Syst. Appl..

[11]  Yasuhiro Hayashi,et al.  Preliminary Analysis of Short-term Solar Irradiance Forecasting by using Total-sky Imager and Convolutional Neural Network , 2019, 2019 IEEE PES GTD Grand International Conference and Exposition Asia (GTD Asia).

[12]  Tat-Seng Chua,et al.  SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Chunheng Wang,et al.  A Selection Criterion for the Optimal Resolution of Ground-Based Remote Sensing Cloud Images for Cloud Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Pietro Perona,et al.  On the usefulness of attention for object recognition , 2004 .

[15]  Carlos D. Castillo,et al.  Deep Heterogeneous Feature Fusion for Template-Based Face Recognition , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[16]  Jun Fu,et al.  Dual Attention Network for Scene Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Hongxun Yao,et al.  Deep Feature Fusion for VHR Remote Sensing Scene Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  R. Pitz-Paal,et al.  Determination of cloud transmittance for all sky imager based solar nowcasting , 2019, Solar Energy.

[20]  George Economou,et al.  A local binary pattern classification approach for cloud types derived from all-sky imagers , 2018, International Journal of Remote Sensing.

[21]  Stefan Winkler,et al.  Categorization of cloud image patches using an improved texton-based approach , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[22]  Wei Huang,et al.  Cloud detection for high-resolution remote-sensing images of urban areas using colour and edge features based on dual-colour models , 2018 .

[23]  Zhiguo Cao,et al.  DeepCloud: Ground-Based Cloud Image Categorization Using Deep Convolutional Features , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[24]  V. McNeill,et al.  Atmospheric Aerosols: Clouds, Chemistry, and Climate. , 2017, Annual review of chemical and biomolecular engineering.

[25]  S. Klein,et al.  Impact of decadal cloud variations on the Earth/'s energy budget , 2016 .

[26]  Chunheng Wang,et al.  Deep Convolutional Activations-Based Features for Ground-Based Cloud Classification , 2017, IEEE Geoscience and Remote Sensing Letters.

[27]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[28]  Peng Zhang,et al.  Affective video content analysis based on multimodal data fusion in heterogeneous networks , 2019, Inf. Fusion.

[29]  Shu Duan,et al.  Cloud Classification and Distribution of Cloud Types in Beijing Using Ka-Band Radar Data , 2019, Advances in Atmospheric Sciences.

[30]  Thomas Peter,et al.  Small-scale cloud processes and climate , 2008, Nature.

[31]  Sam Kwong,et al.  G-MS2F: GoogLeNet based multi-stage feature fusion of deep CNN for scene recognition , 2017, Neurocomputing.

[32]  Dennis L. Hartmann,et al.  Clouds and the Atmospheric Circulation Response to Warming , 2016 .

[33]  Yuhan Tang,et al.  Features of the Cloud Base Height and Determining the Threshold of Relative Humidity over Southeast China , 2019, Remote. Sens..

[34]  Yoshua Bengio,et al.  Attention-Based Models for Speech Recognition , 2015, NIPS.

[35]  Yuxin Peng,et al.  Object-Part Attention Model for Fine-Grained Image Classification , 2017, IEEE Transactions on Image Processing.

[36]  Yuhao Wang,et al.  Real-Time Dense Semantic Labeling with Dual-Path Framework for High-Resolution Remote Sensing Image , 2019, Remote. Sens..

[37]  Zhiguo Cao,et al.  mCLOUD: A Multiview Visual Feature Extraction Mechanism for Ground-Based Cloud Image Categorization , 2016 .

[38]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Li Fei-Fei,et al.  Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Zheru Chi,et al.  Facial Expression Recognition in Video with Multiple Feature Fusion , 2018, IEEE Transactions on Affective Computing.

[42]  Mei Li,et al.  Deep multimodal fusion for ground-based cloud classification in weather station networks , 2018, EURASIP J. Wirel. Commun. Netw..

[43]  Christopher D Cappa,et al.  Atmospheric processes and their controlling influence on cloud condensation nuclei activity. , 2015, Chemical reviews.

[44]  Junseok Kwon,et al.  Deep Meta Learning for Real-Time Target-Aware Visual Tracking , 2017, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[45]  Shutao Li,et al.  Hyperspectral Image Classification With Deep Feature Fusion Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Xiuping Jia,et al.  Deep Fusion of Remote Sensing Data for Accurate Classification , 2017, IEEE Geoscience and Remote Sensing Letters.

[47]  Jun Yang,et al.  Cloud Type Classification of Total-Sky Images Using Duplex Norm-Bounded Sparse Coding , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[48]  Robert Pitz-Paal,et al.  Cloud height and tracking accuracy of three all sky imager systems for individual clouds , 2019, Solar Energy.

[49]  Hsu-Yung Cheng,et al.  Multi-model solar irradiance prediction based on automatic cloud classification , 2015 .

[50]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[51]  Feng Zhang,et al.  CloudNet: Ground‐Based Cloud Classification With Deep Convolutional Neural Network , 2018, Geophysical Research Letters.

[52]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[53]  Hanqing Lu,et al.  Attention CoupleNet: Fully Convolutional Attention Coupling Network for Object Detection , 2019, IEEE Transactions on Image Processing.

[54]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  Baihua Xiao,et al.  Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network , 2018, Remote. Sens..

[56]  Xiaogang Wang,et al.  Residual Attention Network for Image Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Yishu Liu,et al.  Scene Classification Based on Two-Stage Deep Feature Fusion , 2018, IEEE Geoscience and Remote Sensing Letters.

[58]  Dong Yu,et al.  Monaural Multi-Talker Speech Recognition with Attention Mechanism and Gated Convolutional Networks , 2018, INTERSPEECH.

[59]  Josep Calbó,et al.  Feature Extraction from Whole-Sky Ground-Based Images for Cloud-Type Recognition , 2008 .

[60]  Hsu-Yung Cheng,et al.  Block-based cloud classification with statistical features and distribution of local texture features , 2014 .

[61]  Shuang Liu,et al.  Hierarchical Multimodal Fusion for Ground-Based Cloud Classification in Weather Station Networks , 2019, IEEE Access.

[62]  Jahan Kariyeva,et al.  Comparing Deep Learning and Shallow Learning for Large-Scale Wetland Classification in Alberta, Canada , 2019, Remote. Sens..

[63]  Jun Yang,et al.  From pixels to patches: a cloud classification method based on a bag of micro-structures , 2015 .

[64]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[65]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.