Reconstruction of turbulent data with deep generative models for semantic inpainting from TURB-Rot database

We study the applicability of tools developed by the computer vision community for features learning and semantic image inpainting to perform data reconstruction of fluid turbulence configurations. The aim is twofold. First, we explore on a quantitative basis, the capability of Convolutional Neural Networks embedded in a Deep Generative Adversarial Model (Deep-GAN) to generate missing data in turbulence, a paradigmatic high dimensional chaotic system. In particular, we investigate their use in reconstructing two-dimensional damaged snapshots extracted from a large database of numerical configurations of 3d turbulence in the presence of rotation, a case with multi-scale random features where both large-scale organised structures and small-scale highly intermittent and non-Gaussian fluctuations are present. Second, following a reverse engineering approach, we aim to rank the input flow properties (features) in terms of their qualitative and quantitative importance to obtain a better set of reconstructed fields. We present two approaches both based on Context Encoders. The first one infers the missing data via a minimization of the L2 pixel-wise reconstruction loss, plus a small adversarial penalisation. The second searches for the closest encoding of the corrupted flow configuration from a previously trained generator. Finally, we present a comparison with a different data assimilation tool, based on Nudging, an equation-informed unbiased protocol, well known in the numerical weather prediction community. The TURB-Rot database, \url{this http URL}, of roughly 300K 2d turbulent images is released and details on how to download it are given.

[1]  J. Hoke,et al.  The Initialization of Numerical Models by a Dynamic-Initialization Technique , 1976 .

[2]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[3]  P. Holmes,et al.  The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows , 1993 .

[4]  Leslie M. Smith,et al.  Transfer of energy to two-dimensional large scales in forced, rotating three-dimensional turbulence , 1999 .

[5]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[6]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[7]  C. Meneveau,et al.  Scale-Invariance and Turbulence Models for Large-Eddy Simulation , 2000 .

[8]  Roger Woodard,et al.  Interpolation of Spatial Data: Some Theory for Kriging , 1999, Technometrics.

[9]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[10]  Eugenia Kalnay,et al.  Atmospheric Modeling, Data Assimilation and Predictability , 2002 .

[11]  Tony F. Chan,et al.  Mathematical Models for Local Nontexture Inpaintings , 2002, SIAM J. Appl. Math..

[12]  A. Mcnally,et al.  A cloud detection algorithm for high‐spectral‐resolution infrared sounders , 2003 .

[13]  Nicole A. Lazar,et al.  Statistical Analysis With Missing Data , 2003, Technometrics.

[14]  Daniele Venturi,et al.  Gappy data and reconstruction procedures for flow past a cylinder , 2004, Journal of Fluid Mechanics.

[15]  Wotao Yin,et al.  An Iterative Regularization Method for Total Variation-Based Image Restoration , 2005, Multiscale Model. Simul..

[16]  George E. Karniadakis,et al.  Gappy data: To Krig or not to Krig? , 2006, J. Comput. Phys..

[17]  G. Evensen Data Assimilation: The Ensemble Kalman Filter , 2006 .

[18]  Nathan E. Murray,et al.  An application of Gappy POD , 2006 .

[19]  P. Sagaut,et al.  Homogeneous Turbulence Dynamics , 2008 .

[20]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[21]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[22]  Evelyn Lunasin,et al.  The Navier–Stokes–Voight model for image inpainting , 2009, 0901.4548.

[23]  Eberhard Bodenschatz,et al.  Limitations of accuracy in PIV due to individual variations of particle image intensities , 2009 .

[24]  Peter Bauer,et al.  Direct 4D‐Var assimilation of all‐sky radiances. Part I: Implementation , 2010 .

[25]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[26]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[27]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[28]  S. Lakshmivarahan,et al.  Nudging Methods: A Critical Overview , 2013 .

[29]  Charles Meneveau,et al.  Synchronization of chaos in fully developed turbulence. , 2012, Physical review letters.

[30]  P. Davidson Turbulence in Rotating, Stratified and Electrically Conducting Fluids , 2013 .

[31]  A. Pouquet,et al.  Geophysical turbulence and the duality of the energy flow across scales. , 2013, Physical review letters.

[32]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[34]  Jonathan Krause,et al.  3D Object Representations for Fine-Grained Categorization , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[35]  E. Sharon,et al.  Experimental observation of steady inertial wave turbulence in deep rotating flows , 2014, Nature Physics.

[36]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[37]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[38]  F. Moisy,et al.  Direct and inverse energy cascades in a forced rotating turbulence experiment , 2014, 1412.3933.

[39]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[40]  Peter Bauer,et al.  The quiet revolution of numerical weather prediction , 2015, Nature.

[41]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[42]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[43]  Edriss S. Titi,et al.  Abridged Continuous Data Assimilation for the 2D Navier–Stokes Equations Utilizing Measurements of Only One Component of the Velocity Field , 2015, 1504.05978.

[44]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[45]  Edriss S. Titi,et al.  A Computational Study of a Data Assimilation Algorithm for the Two-dimensional Navier-Stokes Equations , 2015, 1505.01234.

[46]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[47]  Fuqing Zhang,et al.  Review of the Ensemble Kalman Filter for Atmospheric Data Assimilation , 2016 .

[48]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  I. Mazzitelli,et al.  Coherent Structures and Extreme Events in Rotating Multiphase Turbulent Flows , 2016, Physical Review X.

[50]  Steven L. Brunton,et al.  Machine Learning Control – Taming Nonlinear Dynamics and Turbulence , 2016, Fluid Mechanics and Its Applications.

[51]  Marc Bocquet,et al.  Data Assimilation: Methods, Algorithms, and Applications , 2016 .

[52]  Carola-Bibiane Schönlieb,et al.  Partial Differential Equation Methods for Image Inpainting , 2015, Cambridge monographs on applied and computational mathematics.

[53]  Katherine L. Milkman,et al.  Should Governments Invest More in Nudging? , 2017, Psychological science.

[54]  Luca Biferale,et al.  Energy transfer in turbulence under rotation. , 2017, 1711.07054.

[55]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  A. Mazzino,et al.  Inferring flow parameters and turbulent configuration with physics-informed data assimilation and spectral nudging , 2018, Physical Review Fluids.

[57]  Luca Biferale,et al.  Cascades and transitions in turbulent flows , 2018, Physics Reports.

[58]  Steven L. Brunton,et al.  Deep learning for universal linear embeddings of nonlinear dynamics , 2017, Nature Communications.

[59]  Unraveling turbulence via physics-informed data-assimilation and spectral nudging , 2018 .

[60]  G. Evensen,et al.  Data assimilation in the geosciences: An overview of methods, issues, and perspectives , 2017, WIREs Climate Change.

[61]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[62]  Vincent Mons,et al.  Kriging-enhanced ensemble variational data assimilation for scalar-source identification in turbulent environments , 2019, J. Comput. Phys..

[63]  Michael W. Mahoney,et al.  Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction , 2019, ArXiv.

[64]  Ray W. Grout,et al.  Data recovery in computational fluid dynamics through deep image priors , 2019 .

[65]  K. Kashinath,et al.  Towards Physics-informed Deep Learning for Spatiotemporal Modeling of Turbulent Flows , 2019 .

[66]  Steven L. Brunton,et al.  Data-Driven Science and Engineering , 2019 .

[67]  Tamer A. Zaki,et al.  Spatial reconstruction of steady scalar sources from remote measurements in turbulent flow , 2019, Journal of Fluid Mechanics.

[68]  A. Mohan,et al.  Compressed Convolutional LSTM: An Efficient Deep Learning framework to Model High Fidelity 3D Turbulence , 2019, 1903.00033.

[69]  Alexander LeNail,et al.  NN-SVG: Publication-Ready Neural Network Architecture Schematics , 2019, J. Open Source Softw..

[70]  Koji Fukagata,et al.  Synthetic turbulent inflow generator using machine learning , 2018, Physical Review Fluids.

[71]  K. Taira,et al.  Super-resolution reconstruction of turbulent flows with machine learning , 2018, Journal of Fluid Mechanics.

[72]  Joachim Denzler,et al.  Deep learning and process understanding for data-driven Earth system science , 2019, Nature.

[73]  Ahmet Gunduz,et al.  Resource Efficient 3D Convolutional Neural Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[74]  Markus H. Gross,et al.  Deep Fluids: A Generative Network for Parameterized Fluid Simulations , 2018, Comput. Graph. Forum.

[75]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[76]  Prabhat,et al.  Highly-Ccalable, Physics-Informed GANs for Learning Solutions of Stochastic PDEs , 2019, 2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS).

[77]  Nils Thürey,et al.  Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow , 2018, Comput. Graph. Forum.

[78]  Thomas Peters,et al.  Data-driven science and engineering: machine learning, dynamical systems, and control , 2019, Contemporary Physics.

[79]  Steven L. Brunton,et al.  Robust flow reconstruction from limited measurements via sparse representation , 2018, Physical Review Fluids.

[80]  Michael Chertkov,et al.  Embedding Hard Physical Constraints in Convolutional Neural Networks for 3D Turbulence , 2020, ICLR 2020.

[81]  Petros Koumoutsakos,et al.  Machine Learning for Fluid Mechanics , 2019, Annual Review of Fluid Mechanics.

[82]  Shady E. Ahmed,et al.  Long short-term memory embedded nudging schemes for nonlinear data assimilation of geophysical flows , 2020, Physics of Fluids.

[83]  Jaideep Pathak,et al.  Combining Machine Learning with Knowledge-Based Modeling for Scalable Forecasting and Subgrid-Scale Closure of Large, Complex, Spatiotemporal Systems , 2020, Chaos.

[84]  K. Carlson,et al.  Turbulent Flows , 2020, Finite Analytic Method in Flows and Heat Transfer.

[85]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[86]  L. Biferale,et al.  Synchronization to Big Data: Nudging the Navier-Stokes Equations for Data Assimilation of Turbulent Flows , 2019, Physical Review X.

[87]  Karthik Kashinath,et al.  Enforcing Physical Constraints in CNNs through Differentiable PDE Layer , 2020, ICLR 2020.

[88]  L. Biferale,et al.  TURB-Rot. A large database of 3d and 2d snapshots from turbulent rotating flows , 2020, ArXiv.

[89]  A. Farhat,et al.  Data Assimilation in Large Prandtl Rayleigh-Bénard Convection from Thermal Measurements , 2019, SIAM J. Appl. Dyn. Syst..

[90]  Steven L. Brunton,et al.  Robust principal component analysis for modal decomposition of corrupt fluid flows , 2019, Physical Review Fluids.

[91]  Marc Bocquet,et al.  Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model , 2019, J. Comput. Sci..

[92]  Karthik Kashinath,et al.  Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems , 2019, J. Comput. Phys..