Exploiting Spatial–Temporal Dynamics for Satellite Image Sequence Prediction

Satellite image sequence prediction is a challenging and significant task. The existing deep learning methods for the task make predictions mainly based on low-level pixelwise features, which fail to model the sophisticated spatial–temporal features of satellite image sequences and deliver unsatisfactory performance. In this letter, we present a hierarchical spatial–temporal network (HSTnet) for satellite image sequence prediction. With a carefully designed hierarchical feature extraction mechanism, HSTnet can learn effective spatial–temporal features from both pixel level and patch level. In addition, to better capture patch-level spatial–temporal dynamics, a dual-branch Transformer is proposed to model patch-level spatial and temporal features, respectively. Comprehensive experiments on the Fengyun-4A (FY-4A) satellite dataset demonstrate the superiority and effectiveness of our proposed method HSTnet over state-of-the-art approaches.

[1]  K. Azizzadenesheli,et al.  FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators , 2022, ArXiv.

[2]  Mingsheng Long,et al.  MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Yue Wu,et al.  A Classified Adversarial Network for Multi-Spectral Remote Sensing Image Change Detection , 2020, Remote. Sens..

[4]  Maomao Li,et al.  Self-Attention ConvLSTM for Spatiotemporal Prediction , 2020, AAAI.

[5]  Jason Hickey,et al.  MetNet: A Neural Weather Model for Precipitation Forecasting , 2020, ArXiv.

[6]  Nicolas Thome,et al.  Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Yong Man Ro,et al.  MCSIP Net: Multichannel Satellite Image Prediction via Deep Neural Network , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Menglong Yan,et al.  Ground Moving Target Indication Based on Optical Flow in Single-Channel SAR , 2019, IEEE Geoscience and Remote Sensing Letters.

[9]  Thomas Blaschke,et al.  Evaluation of Different Machine Learning Methods and Deep-Learning Convolutional Neural Networks for Landslide Detection , 2019, Remote. Sens..

[10]  Philip S. Yu,et al.  Memory in Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity From Spatiotemporal Dynamics , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Philip S. Yu,et al.  PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning , 2018, ICML.

[12]  Philip S. Yu,et al.  PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs , 2017, NIPS.

[13]  Christopher Joseph Pal,et al.  Delving Deeper into Convolutional Networks for Learning Video Representations , 2015, ICLR.

[14]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[15]  Pradip Kumar Pal,et al.  Extrapolation of Sequence of Geostationary Satellite Images for Weather Nowcasting , 2011, IEEE Geoscience and Remote Sensing Letters.

[16]  K. Tan,et al.  Evolutionary Multiform Optimization With Two-Stage Bidirectional Knowledge Transfer Strategy for Point Cloud Registration , 2024, IEEE Transactions on Evolutionary Computation.

[17]  Xutao Li,et al.  MSTCGAN: Multi-Scale Time Conditional Generative Adversarial Network for Long-term Satellite Image Sequence Prediction , 2022, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Le Hoang Son,et al.  Some novel hybrid forecast methods based on picture fuzzy clustering for weather nowcasting from satellite image sequences , 2016, Applied Intelligence.