Attention-based Convolutional Autoencoders for 3D-Variational Data Assimilation

Abstract We propose a new ‘Bi-Reduced Space’ approach to solving 3D Variational Data Assimilation using Convolutional Autoencoders. We prove that our approach has the same solution as previous methods but has significantly lower computational complexity; in other words, we reduce the computational cost without affecting the data assimilation accuracy. We tested our proposal with data from a real-world application: a pollution model of a site in Elephant and Castle (London, UK) and found that we could (1) reduce the size of the background covariance matrix representation by O ( 1 0 3 ) , and (2) increase our data assimilation accuracy with respect to existing reduced space methods.

[1]  Jonathan Tompson,et al.  Efficient object localization using Convolutional Networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[3]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Yun Fu,et al.  Residual Non-local Attention Networks for Image Restoration , 2019, ICLR.

[5]  Lei Zhou,et al.  Variational Autoencoder for Low Bit-rate Image Compression , 2018, CVPR Workshops.

[6]  Zhengdong Lu,et al.  Fast neural network surrogates for very high dimensional physics-based models in computational oceanography , 2007, Neural Networks.

[7]  Thomas S. Huang,et al.  Wide-activated Deep Residual Networks based Restoration for BPG-compressed Images , 2018, CVPR Workshops.

[8]  Hassan Foroosh,et al.  Factorized Convolutional Neural Networks , 2016, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[9]  Qiu Shen,et al.  Extreme Image Coding via Multiscale Autoencoders with Generative Adversarial Optimization , 2019, 2019 IEEE Visual Communications and Image Processing (VCIP).

[10]  Mathew J. Owens,et al.  A Variational Approach to Data Assimilation in the Solar Wind , 2018, Space Weather.

[11]  Andrew C. Lorenc,et al.  A comparison of hybrid variational data assimilation methods for global NWP , 2018, Quarterly Journal of the Royal Meteorological Society.

[12]  Zhengfang Duanmu,et al.  End-to-End Blind Image Quality Assessment Using Deep Neural Networks , 2018, IEEE Transactions on Image Processing.

[13]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[14]  Seunghyun Cho.  Low Bit-rate Image Compression based on Post-processing with Grouped Residual Dense Network , 2019, CVPR Workshops.

[15]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[16]  Luc Van Gool,et al.  Conditional Probability Models for Deep Image Compression , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Wei Huang,et al.  A Three-Dimensional Variational Data Assimilation System for MM5: Implementation and Initial Results , 2004 .

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[20]  Zhe Gan,et al.  Variational Autoencoder for Deep Learning of Images, Labels and Captions , 2016, NIPS.

[21]  C. Pain,et al.  Model identification of reduced order fluid dynamics systems using deep learning , 2017, International Journal for Numerical Methods in Fluids.

[22]  Danna Zhou,et al.  d. , 1840, Microbial pathogenesis.

[23]  R. Bannister.  A review of operational methods of variational and ensemble‐variational data assimilation , 2017 .

[24]  Dong-Wook Kim,et al.  GRDN:Grouped Residual Dense Network for Real Image Denoising and GAN-Based Real-World Noise Modeling , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Nassir Navab,et al.  Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images , 2018, BrainLes@MICCAI.

[26]  John Derber,et al.  The National Meteorological Center's spectral-statistical interpolation analysis system , 1992 .

[27]  D. Zupanski.  A General Weak Constraint Applicable to Operational 4DVAR Data Assimilation Systems , 1997 .

[28]  Jason Cong,et al.  Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks , 2015, FPGA.

[29]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[30]  Juan Du,et al.  Parameterised non-intrusive reduced order methods for ensemble Kalman filter data assimilation , 2018, Computers & Fluids.

[31]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Cédric Jamet,et al.  Data Assimilation Methods , 2013 .

[33]  Andrew C. Lorenc,et al.  Analysis methods for numerical weather prediction , 1986 .

[34]  Etienne Arbogast,et al.  A 3D ensemble variational data assimilation scheme for the limited‐area AROME model: Formulation and preliminary results , 2018, Quarterly Journal of the Royal Meteorological Society.

[35]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Adrian Sandu,et al.  Four-dimensional data assimilation experiments with International Consortium for Atmospheric Research on Transport and Transformation ozone measurements , 2007 .

[37]  英樹 麻生.  深層学習（Deep Learning）をめぐって , 2017 .

[38]  A. Robins,et al.  Enhancing CFD-LES air pollution prediction accuracy using data assimilation , 2019, Building and Environment.

[39]  Ming Li.  VimicroABCnet: An Image Coder Combining A Better Color Space Conversion Algorithm and A Post Enhancing Network , 2019, CVPR Workshops.

[40]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[41]  Almerico Murli,et al.  On the variational data assimilation problem solving and sensitivity analysis , 2017, J. Comput. Phys..

[42]  Jing Zhou,et al.  Multi-scale and Context-adaptive Entropy Model for Image Compression , 2019, CVPR Workshops.

[43]  A. Lorenc.  Optimal nonlinear objective analysis , 1988 .

[44]  Soumik Sarkar,et al.  LLNet: A deep autoencoder approach to natural low-light image enhancement , 2015, Pattern Recognit..

[45]  Takehisa Yairi,et al.  Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction , 2014, MLSDA'14.

[46]  Carl Doersch,et al.  Tutorial on Variational Autoencoders , 2016, ArXiv.

[47]  Lucas Theis,et al.  Lossy Image Compression with Compressive Autoencoders , 2017, ICLR.

[48]  Pejman Shoeibi Omrani,et al.  Deep Learning and Data Assimilation for Real-Time Production Prediction in Natural Gas Wells , 2018, ArXiv.

[49]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Valero Laparra,et al.  Density Modeling of Images using a Generalized Normalization Transformation , 2015, ICLR.

[51]  Zhan Ma,et al.  Learned Image Restoration for VVC Intra Coding , 2019, CVPR Workshops.

[52]  Jiro Katto,et al.  Deep Convolutional AutoEncoder-based Lossy Image Compression , 2018, 2018 Picture Coding Symposium (PCS).

[53]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[54]  Adrian Sandu,et al.  A hybrid approach to estimating error covariances in variational data assimilation , 2010 .

[55]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Hanan Samet,et al.  Pruning Filters for Efficient ConvNets , 2016, ICLR.

[57]  P. Courtier,et al.  The ECMWF implementation of three‐dimensional variational assimilation (3D‐Var). II: Structure functions , 1998 .

[58]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[60]  Yike Guo,et al.  Effective variational data assimilation in air-pollution prediction , 2018, Big Data Min. Anal..

[61]  A. K. Cline,et al.  Computation of the Singular Value Decomposition , 2006 .

[62]  Ionel M. Navon,et al.  An efficient goal‐based reduced order model approach for targeted adaptive observations , 2017 .

[63]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[64]  Lei Zhou,et al.  End-to-end Optimized Image Compression with Attention Mechanism , 2019, CVPR Workshops.

[65]  Yan Zhou,et al.  CNN-Optimized Image Compression with Uncertainty based Resource Allocation , 2018, CVPR Workshops.

[66]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[67]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[68]  A. Chun,et al.  On the brain , 2007, Nature Nanotechnology.

[69]  David Zhang,et al.  Learning Convolutional Networks for Content-Weighted Image Compression , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[70]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[71]  N. Pinardi,et al.  An oceanographic three-dimensional variational data assimilation scheme , 2008 .

[72]  Bo Chen,et al.  Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[73]  P. Courtier,et al.  The ECMWF implementation of three‐dimensional variational assimilation (3D‐Var). I: Formulation , 1998 .

[74]  Dario Grana,et al.  Ensemble-based seismic history matching with data reparameterization using convolutional autoencoder , 2018, SEG Technical Program Expanded Abstracts 2018.

[75]  Zhan Ma,et al.  Extreme Image Compression via Multiscale Autoencoders With Generative Adversarial Optimization , 2019, ArXiv.

[76]  David Minnen,et al.  Variational image compression with a scale hyperprior , 2018, ICLR.

[77]  Xin Chen,et al.  Deep Learning-Based Model Reduction for Distributed Parameter Systems , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[78]  David Kappel,et al.  Deep Rewiring: Training very sparse deep networks , 2017, ICLR.

[79]  Joseph Tribbia,et al.  Scale Interactions and Atmospheric Predictability: An Updated Perspective , 2004 .

[80]  R. Giryes,et al.  Autoencoders , 2021, Deep Learning in Science.

[81]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[82]  P. Courtier,et al.  A strategy for operational implementation of 4D‐Var, using an incremental approach , 1994 .

[83]  Matt J. Kusner,et al.  Grammar Variational Autoencoder , 2017, ICML.

[84]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[85]  Christopher C. Pain,et al.  Optimal reduced space for Variational Data Assimilation , 2019, J. Comput. Phys..

[86]  Yun Fu,et al.  Residual Dense Network for Image Restoration , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.