Reconstruction of turbulent data with deep generative models for semantic inpainting from TURB-Rot database

We study the applicability of tools developed by the computer vision community for features learning and semantic image inpainting to perform data reconstruction of fluid turbulence configurations. The aim is twofold. First, we explore on a quantitative basis, the capability of Convolutional Neural Networks embedded in a Deep Generative Adversarial Model (Deep-GAN) to generate missing data in turbulence, a paradigmatic high dimensional chaotic system. In particular, we investigate their use in reconstructing two-dimensional damaged snapshots extracted from a large database of numerical configurations of 3d turbulence in the presence of rotation, a case with multi-scale random features where both large-scale organised structures and small-scale highly intermittent and non-Gaussian fluctuations are present. Second, following a reverse engineering approach, we aim to rank the input flow properties (features) in terms of their qualitative and quantitative importance to obtain a better set of reconstructed fields. We present two approaches both based on Context Encoders. The first one infers the missing data via a minimization of the L2 pixel-wise reconstruction loss, plus a small adversarial penalisation. The second searches for the closest encoding of the corrupted flow configuration from a previously trained generator. Finally, we present a comparison with a different data assimilation tool, based on Nudging, an equation-informed unbiased protocol, well known in the numerical weather prediction community. The TURB-Rot database, \url{this http URL}, of roughly 300K 2d turbulent images is released and details on how to download it are given.

[1]  K. Taira,et al.  Super-resolution reconstruction of turbulent flows with machine learning , 2018, Journal of Fluid Mechanics.

[2]  Koji Fukagata,et al.  Synthetic turbulent inflow generator using machine learning , 2018, Physical Review Fluids.

[3]  Vincent Mons,et al.  Kriging-enhanced ensemble variational data assimilation for scalar-source identification in turbulent environments , 2019, J. Comput. Phys..

[4]  Carola-Bibiane Schönlieb,et al.  Partial Differential Equation Methods for Image Inpainting , 2015, Cambridge monographs on applied and computational mathematics.

[5]  Steven L. Brunton,et al.  Machine Learning Control – Taming Nonlinear Dynamics and Turbulence , 2016, Fluid Mechanics and Its Applications.

[6]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[7]  Omer San,et al.  Long short-term memory embedded nudging schemes for nonlinear data assimilation of geophysical flows , 2020 .

[8]  A. Mohan,et al.  Compressed Convolutional LSTM: An Efficient Deep Learning framework to Model High Fidelity 3D Turbulence , 2019, 1903.00033.

[9]  Leslie M. Smith,et al.  Transfer of energy to two-dimensional large scales in forced, rotating three-dimensional turbulence , 1999 .

[10]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[11]  S. Lakshmivarahan,et al.  Nudging Methods: A Critical Overview , 2013 .

[12]  Karthik Kashinath,et al.  Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems , 2019, J. Comput. Phys..

[13]  Marc Bocquet,et al.  Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model , 2019, J. Comput. Sci..

[14]  A. Pouquet,et al.  Geophysical turbulence and the duality of the energy flow across scales. , 2013, Physical review letters.

[15]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[16]  I. Mazzitelli,et al.  Coherent Structures and Extreme Events in Rotating Multiphase Turbulent Flows , 2016, Physical Review X.

[17]  Eberhard Bodenschatz,et al.  Limitations of accuracy in PIV due to individual variations of particle image intensities , 2009 .

[18]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[19]  A. Mcnally,et al.  A cloud detection algorithm for high‐spectral‐resolution infrared sounders , 2003 .

[20]  Prabhat,et al.  Highly-Ccalable, Physics-Informed GANs for Learning Solutions of Stochastic PDEs , 2019, 2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS).

[21]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[22]  K. Carlson,et al.  Turbulent Flows , 2020, Finite Analytic Method in Flows and Heat Transfer.

[23]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[24]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[25]  Ray W. Grout,et al.  Data recovery in computational fluid dynamics through deep image priors , 2019 .

[26]  Jonathan Krause,et al.  3D Object Representations for Fine-Grained Categorization , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[27]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[28]  Katherine L. Milkman,et al.  Should Governments Invest More in Nudging? , 2017, Psychological science.

[29]  Karthik Kashinath,et al.  Enforcing Physical Constraints in CNNs through Differentiable PDE Layer , 2020, ICLR 2020.

[30]  Tamer A. Zaki,et al.  Spatial reconstruction of steady scalar sources from remote measurements in turbulent flow , 2019, Journal of Fluid Mechanics.

[31]  J. Hoke,et al.  The Initialization of Numerical Models by a Dynamic-Initialization Technique , 1976 .

[32]  Nicole A. Lazar,et al.  Statistical Analysis With Missing Data , 2003, Technometrics.

[33]  Luca Biferale,et al.  Cascades and transitions in turbulent flows , 2018, Physics Reports.

[34]  Markus H. Gross,et al.  Deep Fluids: A Generative Network for Parameterized Fluid Simulations , 2018, Comput. Graph. Forum.

[35]  Steven L. Brunton,et al.  Deep learning for universal linear embeddings of nonlinear dynamics , 2017, Nature Communications.

[36]  Nils Thürey,et al.  Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow , 2018, Comput. Graph. Forum.

[37]  L. Biferale,et al.  Synchronization to Big Data: Nudging the Navier-Stokes Equations for Data Assimilation of Turbulent Flows , 2019, Physical Review X.

[38]  Peter Bauer,et al.  The quiet revolution of numerical weather prediction , 2015, Nature.

[39]  Steven L. Brunton,et al.  Data-Driven Science and Engineering , 2019 .

[40]  Tony F. Chan,et al.  Mathematical Models for Local Nontexture Inpaintings , 2002, SIAM J. Appl. Math..

[41]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[42]  Roger Woodard,et al.  Interpolation of Spatial Data: Some Theory for Kriging , 1999, Technometrics.

[43]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[44]  Daniele Venturi,et al.  Gappy data and reconstruction procedures for flow past a cylinder , 2004, Journal of Fluid Mechanics.

[45]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[46]  Petros Koumoutsakos,et al.  Machine Learning for Fluid Mechanics , 2019, Annual Review of Fluid Mechanics.

[47]  Fuqing Zhang,et al.  Review of the Ensemble Kalman Filter for Atmospheric Data Assimilation , 2016 .

[48]  A. Mazzino,et al.  Inferring flow parameters and turbulent configuration with physics-informed data assimilation and spectral nudging , 2018, Physical Review Fluids.

[49]  Jaideep Pathak,et al.  Combining Machine Learning with Knowledge-Based Modeling for Scalable Forecasting and Subgrid-Scale Closure of Large, Complex, Spatiotemporal Systems , 2020, Chaos.

[50]  Ahmet Gunduz,et al.  Resource Efficient 3D Convolutional Neural Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[51]  Michael W. Mahoney,et al.  Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction , 2019, ArXiv.

[52]  Wotao Yin,et al.  An Iterative Regularization Method for Total Variation-Based Image Restoration , 2005, Multiscale Model. Simul..

[53]  Evelyn Lunasin,et al.  The Navier–Stokes–Voight model for image inpainting , 2009, 0901.4548.

[54]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[55]  Peter Bauer,et al.  Direct 4D‐Var assimilation of all‐sky radiances. Part I: Implementation , 2010 .

[56]  L. Biferale,et al.  TURB-Rot. A large database of 3d and 2d snapshots from turbulent rotating flows , 2020, ArXiv.

[57]  Alexander LeNail,et al.  NN-SVG: Publication-Ready Neural Network Architecture Schematics , 2019, J. Open Source Softw..

[58]  Edriss S. Titi,et al.  Abridged Continuous Data Assimilation for the 2D Navier–Stokes Equations Utilizing Measurements of Only One Component of the Velocity Field , 2015, 1504.05978.

[59]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[60]  P. Sagaut,et al.  Homogeneous Turbulence Dynamics , 2008 .

[61]  P. Davidson Turbulence in Rotating, Stratified and Electrically Conducting Fluids , 2013 .

[62]  G. Evensen,et al.  Data assimilation in the geosciences: An overview of methods, issues, and perspectives , 2017, WIREs Climate Change.

[63]  Marc Bocquet,et al.  Data Assimilation: Methods, Algorithms, and Applications , 2016 .

[64]  Thomas Peters,et al.  Data-driven science and engineering: machine learning, dynamical systems, and control , 2019, Contemporary Physics.

[65]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[66]  F. Moisy,et al.  Direct and inverse energy cascades in a forced rotating turbulence experiment , 2014, 1412.3933.

[67]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[68]  Luca Biferale,et al.  Energy transfer in turbulence under rotation. , 2017, 1711.07054.

[69]  Steven L. Brunton,et al.  Robust principal component analysis for modal decomposition of corrupt fluid flows , 2019, Physical Review Fluids.

[70]  Edriss S. Titi,et al.  A Computational Study of a Data Assimilation Algorithm for the Two-dimensional Navier-Stokes Equations , 2015, 1505.01234.

[71]  Steven L. Brunton,et al.  Robust flow reconstruction from limited measurements via sparse representation , 2018, Physical Review Fluids.

[72]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[73]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[74]  Nathan E. Murray,et al.  An application of Gappy POD , 2006 .

[75]  Eugenia Kalnay,et al.  Atmospheric Modeling, Data Assimilation and Predictability , 2002 .

[76]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  Charles Meneveau,et al.  Synchronization of chaos in fully developed turbulence. , 2012, Physical review letters.

[78]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Joachim Denzler,et al.  Deep learning and process understanding for data-driven Earth system science , 2019, Nature.

[80]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[81]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[82]  George E. Karniadakis,et al.  Gappy data: To Krig or not to Krig? , 2006, J. Comput. Phys..

[83]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[84]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[85]  Michael Chertkov,et al.  Embedding Hard Physical Constraints in Convolutional Neural Networks for 3D Turbulence , 2020, ICLR 2020.

[86]  K. Kashinath,et al.  Towards Physics-informed Deep Learning for Spatiotemporal Modeling of Turbulent Flows , 2019 .

[87]  A. Farhat,et al.  Data Assimilation in Large Prandtl Rayleigh-Bénard Convection from Thermal Measurements , 2019, SIAM J. Appl. Dyn. Syst..

[88]  E. Sharon,et al.  Experimental observation of steady inertial wave turbulence in deep rotating flows , 2014, Nature Physics.

[89]  G. Evensen Data Assimilation: The Ensemble Kalman Filter , 2006 .

[90]  C. Meneveau,et al.  Scale-Invariance and Turbulence Models for Large-Eddy Simulation , 2000 .

[91]  P. Holmes,et al.  The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows , 1993 .