Refactoring Scientific Applications for Massive Parallelism

We describe several common problems that we discovered during our efforts to refactor several large geofluid applications that are components of the Community Climate System Model (CCSM) developed at the National Center for Atmospheric Research (NCAR). We stress tested the weak scalability of these applications by studying the impact of increasing both the resolution and core counts by factors of 10–100. Several common code design and implementations issues emerged that prevented the efficient execution of these applications on very large microprocessor counts. We found that these problems arise as a direct result of disparity between the initial design assumptions made for low resolution models running on a few dozen processors, and today’s requirements that applications run in massively parallel computing environments. The issues discussed include non-scalable memory usage and execution time in both the applications themselves and the supporting scientific data tool chains.

[1]  Mariana Vertenstein,et al.  Vectorizing the Community Land Model , 2005, Int. J. High Perform. Comput. Appl..

[2]  Jack Dongarra,et al.  MPI - The Complete Reference: Volume 1, The MPI Core , 1998 .

[3]  John M. Dennis,et al.  Scaling climate simulation applications on the IBM Blue Gene/L system , 2008, IBM J. Res. Dev..

[4]  Ross J. Murray,et al.  Explicit Generation of Orthogonal Grids for Ocean Models , 1996 .

[5]  W. Collins,et al.  The Formulation and Atmospheric Simulation of the Community Atmosphere Model Version 3 (CAM3) , 2006 .

[6]  John M. Dennis.  Inverse Space-Filling Curve Partitioning of a Global Ocean Model , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[7]  Jay Walter Larson,et al.  The Model Coupling Toolkit: A New Fortran90 Toolkit for Building Multiphysics Parallel Coupled Models , 2005, Int. J. High Perform. Comput. Appl..

[8]  William Gropp,et al.  Mpi---the complete reference: volume 1 , 1998 .

[9]  T. Wilbanks,et al.  Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change , 2007 .

[10]  John M. Levesque,et al.  Practical performance portability in the Parallel Ocean Program (POP) , 2005, Concurr. Pract. Exp..

[11]  J. Larson,et al.  M × N Communication and Parallel Interpolation in CCSM 3 Using the Model Coupling Toolkit , 2005 .

[12]  David F. Heidel,et al.  An Overview of the BlueGene/L Supercomputer , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[13]  Mark A. Taylor,et al.  High-Resolution Mesh Convergence Properties and Parallel Efficiency of a Spectral Element Atmospheric Dynamical Core , 2005, Int. J. High Perform. Comput. Appl..

[14]  Robert D. Falgout,et al.  An assumed partition algorithm for determining processor inter-communication , 2006, Parallel Comput..

[15]  Mark A. Taylor,et al.  Petascale atmospheric models for the Community Climate System Model: new developments and evaluation of scalable dynamical cores , 2008 .

[16]  TufoHenry,et al.  High-Resolution Mesh Convergence Properties and Parallel Efficiency of a Spectral Element Atmospheric Dynamical Core , 2005 .

[17]  Jay Walter Larson,et al.  M × N Communication and Parallel Interpolation in Community Climate System Model Version 3 Using the Model Coupling Toolkit , 2005, Int. J. High Perform. Comput. Appl..

[18]  Hiroshi Takahara,et al.  A 26.58 Tflops Global Atmospheric Simulation with the Spectral Transform Method on the Earth Simulator , 2002, ACM/IEEE SC 2002 Conference (SC'02).