MESHFREEFLOWNET: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework

We propose MeshfreeFlowNet, a novel deep learning-based super-resolution framework to generate continuous (grid-free) spatio-temporal solutions from the low-resolution inputs. While being computationally efficient, MeshfreeFlowNet accurately recovers the fine-scale quantities of interest. MeshfreeFlowNet allows for: (i) the output to be sampled at all spatio-temporal resolutions, (ii) a set of Partial Differential Equation (PDE) constraints to be imposed, and (iii) training on fixed-size inputs on arbitrarily sized spatio-temporal domains owing to its fully convolutional encoder. We empirically study the performance of MeshfreeFlowNet on the task of super-resolution of turbulent flows in the Rayleigh-Benard convection problem. Across a diverse set of evaluation metrics, we show that MeshfreeFlowNet significantly outperforms existing baselines. Furthermore, we provide a large scale implementation of MeshfreeFlowNet and show that it efficiently scales across large clusters, achieving 96.80% scaling efficiency on up to 128 GPUs and a training time of less than 4 minutes.

[1]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[2]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[3]  A. Russell,et al.  Multiscale modeling of pollutant transport and chemistry , 1991 .

[4]  Chaoqun Liu,et al.  Multiple scale simulation for transitional and turbulent flow , 1995 .

[5]  G. Eigenberger,et al.  Multiscale modeling of hydrodynamics, mass transfer and reaction in bubble column reactors , 2001 .

[6]  H. Tchelepi,et al.  Multi-scale finite-volume method for elliptic problems in subsurface flow simulation , 2003 .

[7]  D. Yeung,et al.  Super-resolution through neighbor embedding , 2004, CVPR 2004.

[8]  F. Bramkamp,et al.  An adaptive multiscale finite volume solver for unsteady and steady state flow computations , 2004 .

[9]  Jinghai Li,et al.  Multiscale analysis and modeling of multiphase chemical reactors , 2004 .

[10]  Patrick Jenny,et al.  Adaptive Multiscale Finite-Volume Method for Multiphase Flow and Transport in Porous Media , 2005, Multiscale Model. Simul..

[11]  M. Modest.  Multiscale Modeling of Turbulence, Radiation, and Combustion Interactions in Turbulent Flames , 2005 .

[12]  E. Weinan,et al.  Heterogeneous multiscale method for the modeling of complex fluids and micro-fluidics , 2005 .

[13]  Patrick Jenny,et al.  Adaptive fully implicit multi-scale finite-volume method for multi-phase flow and transport in heterogeneous porous media , 2006, J. Comput. Phys..

[14]  H. Tchelepi,et al.  Onset of convection in a gravitationally unstable diffusive boundary layer in porous media , 2005, Journal of Fluid Mechanics.

[15]  Guoqing Hu,et al.  Multiscale phenomena in microfluidics and nanofluidics , 2007 .

[16]  H. Shum,et al.  Image super-resolution using gradient profile prior , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Thomas S. Huang,et al.  Image super-resolution as sparse representation of raw image patches , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  G. Meehl,et al.  A Unified Modeling Approach to Climate System Prediction , 2009 .

[19]  Kai H. Luo,et al.  Multiscale modeling of multiphase flow with complex interactions , 2009 .

[20]  Zhiwei Xiong,et al.  Robust Web Image/Video Super-Resolution , 2010, IEEE Transactions on Image Processing.

[21]  B. Bagchi,et al.  Interplay between multiple length and time scales in complex chemical systems , 2010 .

[22]  Kwang In Kim,et al.  Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[24]  Shaoping Quan,et al.  Simulations of multiphase flows with multiple length scales using moving mesh interface tracking with adaptive meshing , 2011, J. Comput. Phys..

[25]  Raanan Fattal,et al.  Image and video upscaling from local self-examples , 2011, TOGS.

[26]  P. Szymczak,et al.  Interacting length scales in the reactive‐infiltration instability , 2013, 1306.5034.

[27]  Samson Cheung,et al.  Improving NASA's Multiscale Modeling Framework for Tropical Cyclone Climate Study , 2013, Computing in Science & Engineering.

[28]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[29]  Wei Li,et al.  Convolutional Neural Networks for Steady Flow Approximation , 2016, KDD.

[30]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[31]  G. Karniadakis,et al.  Multiscale modeling and simulation of brain blood flow. , 2016, Physics of fluids.

[32]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[33]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[34]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Xin Lin,et al.  Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN , 2017, 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR).

[36]  Yung-Yu Chuang,et al.  Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Björn Ommer,et al.  A Variational U-Net for Conditional Appearance and Shape Generation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[39]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[40]  Georgios Tzimiropoulos,et al.  Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Kyung-Ah Sohn,et al.  Fast, Accurate, and, Lightweight Super-Resolution with Cascading Residual Network , 2018, ECCV.

[42]  Boqiang Liu,et al.  S3D-UNet: Separable 3D U-Net for Brain Tumor Segmentation , 2018, BrainLes@MICCAI.

[43]  Xin Fu,et al.  Artistic Image Generation from Sketch by Using Conditional Adversarial Network and Style Feature Transform , 2018 .

[44]  Yifan Wang,et al.  A Fully Progressive Approach to Single-Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Silvio Savarese,et al.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Xin Chen,et al.  Anime Sketch Coloring with Swish-Gated Residual U-Net , 2018, Communications in Computer and Information Science.

[47]  Yu Song,et al.  A Method for Coloring Low-resolution Black and White Old Movies through Object Understanding , 2019, 2019 Chinese Control And Decision Conference (CCDC).

[48]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[49]  Leah Bar,et al.  Unsupervised Deep Learning Algorithm for PDE-based Forward and Inverse Problems , 2019, ArXiv.

[50]  Karthik Duraisamy,et al.  Prediction of aerodynamic flow fields using convolutional neural networks , 2019, Computational Mechanics.

[51]  Hamed Darabi,et al.  A General Spatio-Temporal Clustering-Based Non-local Formulation for Multiscale Modeling of Compartmentalized Reservoirs , 2019, Day 2 Wed, April 24, 2019.

[52]  D. Castineira,et al.  Multiscale modeling of compartmentalized reservoirs using a hybrid clustering-based non-local approach , 2020 .

[53]  Keaton J. Burns,et al.  Dedalus: A flexible framework for numerical simulations with spectral methods , 2019, Physical Review Research.

[54]  Hamdi A. Tchelepi,et al.  Two-phase multiscale numerical framework for modeling thin films on curved solid surfaces in porous media , 2020, J. Comput. Phys..

[55]  Thomas Funkhouser,et al.  Local Implicit Grid Representations for 3D Scenes , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Karthik Kashinath,et al.  Enforcing Physical Constraints in CNNs through Differentiable PDE Layer , 2020, ICLR 2020.

[57]  Rui Wang,et al.  Towards Physics-informed Deep Learning for Turbulent Flow Prediction , 2019, KDD.

[58]  Kamyar Azizzadenesheli,et al.  Neural Operator: Graph Kernel Network for Partial Differential Equations , 2020, ICLR 2020.

[59]  Zhiming Luo,et al.  A Global and Local Enhanced Residual U-Net for Accurate Retinal Vessel Segmentation , 2019, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[60]  Kamyar Azizzadenesheli,et al.  EikoNet: Solving the Eikonal Equation With Deep Neural Networks , 2020, IEEE Transactions on Geoscience and Remote Sensing.