Insertion of PETSc in the OpenFOAM Framework

OpenFOAM is a widely used open source framework for simulation in several areas of computational fluid dynamics and engineering. As a partial differential equation (PDE)-based framework, OpenFOAM suffers from a performance bottleneck in solving large-scale sparse linear systems of equations. To address the problem, this article proposes a novel OpenFOAM-PETSc framework by inserting PETSc, a dedicated numerical solving package, into the OpenFOAM to speed up the process of solving linear equation systems. The design of the OpenFOAM-PETSc framework is described, and the implementation of an efficient matrix conversion algorithm is given as a case study. Validation tests on a high-performance computing cluster show that OpenFOAM-PETSc reduces the time of solving PDEs by about 27% in the lid-driven cavity flow case and by more than 50% in flow around the cylinder case in comparison with OpenFOAM, without compromising the scalability. In addition, this article also gives a preliminary performance analysis of different numerical solution methods, which may provide guidelines for further optimizations.

[1]  Barbara I. Wohlmuth,et al.  Performance and Scalability of Hierarchical Hybrid Multigrid Solvers for Stokes Systems , 2015, SIAM J. Sci. Comput..

[2]  Murat Manguoglu.  A General Sparse Sparse Linear System Solver and Its Application in OpenFOAM , 2011 .

[3]  Lionel M. Ni,et al.  Another view on parallel speedup , 1990, Proceedings SUPERCOMPUTING '90.

[4]  Massimiliano Culpo.  - ri . eu Partnership for Advanced Computing in Europe Current Bottlenecks in the Scalability of OpenFOAM on Massively Parallel Clusters , 2012 .

[5]  Karl Fürlinger,et al.  Parallel Aspects of OpenFOAM with Large Eddy Simulations , 2011, 2011 IEEE International Conference on High Performance Computing and Communications.

[6]  A. Gosman,et al.  Solution of the implicitly discretised reacting flow equations by operator-splitting , 1986 .

[7]  J. W. Ruge,et al.  4. Algebraic Multigrid , 1987 .

[8]  William Gropp,et al.  Performance Modeling and Tuning of an Unstructured Mesh CFD Application , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[9]  Robert D. Falgout,et al.  hypre: A Library of High Performance Preconditioners , 2002, International Conference on Computational Science.

[10]  William Gropp,et al.  High-performance parallel implicit CFD , 2001, Parallel Comput..

[11]  Martin Schulz,et al.  On the Performance of an Algebraic Multigrid Solver on Multicore Clusters , 2010, VECPAR.

[12]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[13]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[14]  Mottaqiallah Taouil.  A hardware Accelerator for the OpenFOAM Sparse Matrix-Vector Product , 2009 .

[15]  D. Bartuschat.  Algebraic Multigrid , 2007 .

[16]  Yuhua Tang,et al.  An Application-Level Synchronous Checkpoint-Recover Method for Parallel CFD Simulation , 2013, CSE.

[17]  David E. Keyes,et al.  ParCFD 2014 Design and Optimization of OpenFOAM-based CFD Applications for Hybrid and Heterogeneous HPC Platforms , 2018 .

[18]  Yuhua Tang,et al.  Performance analysis of the graph-partitioning algorithms used in OpenFOAM , 2012, 2012 IEEE Fifth International Conference on Advanced Computational Intelligence (ICACI).

[19]  Almerico Murli,et al.  Insertion of PETSc in the NEMO stack software driving NEMO towards exascale computing , 2014, 2014 International Conference on High Performance Computing & Simulation (HPCS).

[20]  Hans De Sterck,et al.  Reducing Complexity in Parallel Algebraic Multigrid Preconditioners , 2004, SIAM J. Matrix Anal. Appl..

[21]  Hasnat Jamil.  Performance Test of PETSc Library for the solution of Fully Coupled Velocity-Pressure Formulation for an Unstructured Finite Volume RANSE Solver , 2010 .