A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations

High performance computing systems are today composed of tens of thousands of processors and deep memory hierarchies. The next generation of machines will further increase the unbalance between I/O capabilities and processing power. To reduce the pressure on I/Os, the in situ analytics paradigm proposes to process the data as closely as possible to where and when the data are produced. Processing can be embedded in the simulation code, executed asynchronously on helper cores on the same nodes, or performed in transit on staging nodes dedicated to analytics. Today, software environnements as well as usage scenarios still need to be investigated before in situ analytics become a standard practice. In this paper we introduce a framework for designing, deploying and executing in situ scenarios. Based on a component model, the scientist designs analytics workflows by first developing processing components that are next assembled in a dataflow graph through a Python script. At runtime the graph is instantiated according to the execution context, the framework taking care of deploying the application on the target architecture and coordinating the analytics workflows with the simulation execution. Component coordination, zero-copy intra-node communications or inter-nodes data transfers rely on per-node distributed daemons. We evaluate various scenarios performing in situ and in transit analytics on large molecular dynamics systems simulated with Gromacs using up to 2048 cores. We show in particular that analytics processing can be performed on the fraction of resources the simulation does not use well, resulting in a limited impact on the simulation performance (less than 9%). Our more advanced scenario combines in situ and in transit processing to compute a molecular surface based on the Quick surf algorithm.

[1]  Karsten Schwan,et al.  PreDatA – preparatory data analytics on peta-scale machines , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[2]  Jeremy S. Meredith,et al.  Parallel in situ coupling of simulation with a fully featured visualization system , 2011, EGPGV '11.

[3]  Robert Sisneros,et al.  Damaris/Viz: A nonintrusive, adaptable and user-friendly in situ visualization framework , 2013, 2013 IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV).

[4]  Jérémie Allard,et al.  Modularity for Large Virtual Reality Applications , 2010, PRESENCE: Teleoperators and Virtual Environments.

[5]  James Arthur Kohl,et al.  Data redistribution and remote method invocation in parallel component architectures , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[6]  Daniel S. Katz,et al.  Pegasus: A framework for mapping complex scientific workflows onto distributed systems , 2005, Sci. Program..

[7]  Fan Zhang,et al.  Enabling In-situ Execution of Coupled Scientific Workflow on Multi-core Platform , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium.

[8]  Carsten Kutzner,et al.  GROMACS 4:  Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation. , 2008, Journal of chemical theory and computation.

[9]  P. Balaji,et al.  GePSeA: A General-Purpose Software Acceleration Framework for Lightweight Task Offloading , 2009, 2009 International Conference on Parallel Processing.

[10]  Oliver Beckstein,et al.  MDAnalysis: A toolkit for the analysis of molecular dynamics simulations , 2011, J. Comput. Chem..

[11]  Kenneth Moreland,et al.  Sandia National Laboratories , 2000 .

[12]  Karsten Schwan,et al.  Just in time: adding value to the IO pipelines of high performance applications with JITStaging , 2011, HPDC '11.

[13]  Karsten Schwan,et al.  Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) , 2008, CLADE '08.

[14]  John L. Klepeis,et al.  A scalable parallel framework for analyzing terascale molecular dynamics simulation trajectories , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[15]  Klaus Schulten,et al.  Mature HIV-1 capsid structure by cryo-electron microscopy and all-atom molecular dynamics , 2013, Nature.

[16]  Klaus Schulten,et al.  Fast Visualization of Gaussian Density Surfaces for Molecular Dynamics and Particle System Trajectories , 2012, EuroVis.

[17]  Marianne Winslett,et al.  High-level buffering for hiding periodic output cost in scientific simulations , 2006, IEEE Transactions on Parallel and Distributed Systems.

[18]  Jae-Yong Lee,et al.  High performance communication between parallel programs , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[19]  Benjamin Lorendeau,et al.  In-Situ visualization in fluid mechanics using Catalyst: A case study for Code Saturne , 2013, 2013 IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV).

[20]  Bertram Ludäscher,et al.  Scientific workflow management and the Kepler system: Research Articles , 2006 .

[21]  Sébastien Limet,et al.  Interactive Molecular Dynamics: Scaling up to Large Systems , 2013, ICCS.

[22]  Joel H. Saltz,et al.  Interoperability of data parallel runtime libraries , 1997, Proceedings 11th International Parallel Processing Symposium.

[23]  Fei Meng,et al.  Functional Partitioning to Optimize End-to-End Performance on Many-core Architectures , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[24]  Fan Zhang,et al.  Combining in-situ and in-transit processing to enable extreme-scale scientific analysis , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[25]  John Biddiscombe,et al.  Parallel computational steering and analysis for HPC applications using a paraview interface and the HDF5 DSM virtual file driver , 2011, EGPGV '11.

[26]  Scott Klasky,et al.  DataSpaces: an interaction and coordination framework for coupled simulation workflows , 2012, HPDC '10.

[27]  Karsten Schwan,et al.  XChange: coupling parallel applications in a dynamic environment , 2004, 2004 IEEE International Conference on Cluster Computing (IEEE Cat. No.04EX935).

[28]  Peter M. Kasson,et al.  GROMACS 4.5: a high-throughput and highly parallel open source molecular simulation toolkit , 2013, Bioinform..

[29]  Scott Klasky,et al.  DART: a substrate for high speed asynchronous data IO , 2008, HPDC '08.

[30]  Karsten Schwan,et al.  FlexIO: I/O Middleware for Location-Flexible Scientific Data Analytics , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[31]  Ray W. Grout,et al.  Ultrascale Visualization In Situ Visualization for Large-Scale Combustion Simulations , 2010 .

[32]  Susan M. Mniszewski,et al.  PAWS: collective interactions and data transfers , 2001, Proceedings 10th IEEE International Symposium on High Performance Distributed Computing.

[33]  Karsten Schwan,et al.  Falcon: On-line monitoring for steering parallel programs , 1998, Concurr. Pract. Exp..

[34]  Edward A. Lee,et al.  Scientific workflow management and the Kepler system , 2006, Concurr. Comput. Pract. Exp..

[35]  Franck Cappello,et al.  Damaris: How to Efficiently Leverage Multicore Parallelism to Achieve Scalable, Jitter-free I/O , 2012, 2012 IEEE International Conference on Cluster Computing.