Language and Runtime Support for Automatic Configuration and Deployment of Scientific Computing Software over Cloud Fabrics

In this paper, we present the design and implementation of Neptune, a simple, domain-specific language based on the Ruby programming language. Neptune automates the configuration and deployment of scientific software frameworks over disparate cloud computing systems. Neptune integrates support for MPI, MapReduce, UPC, X10, StochKit, and others. We implement Neptune as a software overlay for the AppScale cloud platform and extend AppScale with support for elasticity and hybrid execution for scientific computing applications. Neptune imposes no overhead on application execution, yet significantly simplifies the application deployment process, enables portability across cloud systems, and promotes lock-in avoidance by specific cloud vendors.

[1]  Chandra Krintz,et al.  Neptune: a domain specific language for deploying hpc software on cloud platforms , 2011, ScienceCloud '11.

[2]  Richard Wolski,et al.  The Eucalyptus Open-Source Cloud-Computing System , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[3]  Timothy J. Rolfe.  A specimen MPI application: N-Queens in parallel , 2008, SGCS.

[4]  Hiroaki Kitano,et al.  The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models , 2003, Bioinform..

[5]  Tarek A. El-Ghazawi,et al.  UPC Performance and Potential: A NPB Experimental Study , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[6]  Chandra Krintz,et al.  AppScale: Open-Source Platform-as-a-Service , 2012 .

[7]  Guilherme Piegas Koslovski,et al.  Executing Distributed Applications on Virtualized Infrastructures Specified with the VXDL Language and Managed by the HIPerNET Framework , 2009, CloudComp.

[8]  John C. Doyle,et al.  Surviving heat shock: control strategies for robustness and performance. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Interner Bericht.  VAMPIR: Visualization and Analysis of MPI Resources , 1996 .

[10]  Mike Williams,et al.  ERLANG for Concurrent Programming , 1993 .

[11]  Chandra Krintz,et al.  AppScale: Scalable and Open AppEngine Application Development and Deployment , 2009, CloudComp.

[12]  Chandra Krintz,et al.  An Evaluation of Distributed Datastores Using the AppScale Cloud Platform , 2010, 2010 IEEE 3rd International Conference on Cloud Computing.

[13]  Brian Drawert,et al.  The diffusive finite state projection algorithm for efficient simulation of the stochastic reaction-diffusion master equation. , 2010, The Journal of chemical physics.

[14]  Sheng Wu,et al.  StochKit2: software for discrete stochastic simulation of biochemical systems with events , 2011, Bioinform..

[15]  Gabrielle Allen,et al.  The SAGA C++ reference implementation: a milestone toward new high-level grid applications , 2006, SC.

[16]  Anthony Skjellum,et al.  A High-Performance, Portable Implementation of the MPI Message Passing Interface Standard , 1996, Parallel Comput..

[17]  Tao Yang,et al.  Adaptive Two-level Thread Management for Fast MPI Execution on Shared Memory Machines , 1999, ACM/IEEE SC 1999 Conference (SC'99).

[18]  Vivek Sarkar,et al.  X10: an object-oriented approach to non-uniform cluster computing , 2005, OOPSLA '05.

[19]  Chandra Krintz,et al.  Database-Agnostic Transaction Support for Cloud Infrastructures , 2011, 2011 IEEE 4th International Conference on Cloud Computing.

[20]  Joe Armstrong,et al.  Concurrent programming in ERLANG , 1993 .

[21]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[22]  Bernie J Daigle,et al.  Automated estimation of rare event probabilities in biochemical systems. , 2011, The Journal of chemical physics.

[23]  Randy H. Katz,et al.  Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center , 2011, NSDI.