Virtual Clusters in the Jetstream Cloud: A story of elasticized HPC

We discuss our work providing resources for batch computing via the Jetstream cloud, in the form of SLURM clusters. While these are mainly used by science gateways, there have been a few used in the more traditional commandline manner. The flexible nature of these has also lent itself well to educational work, and has provided the basis for a very successful series of tutorials and workshops. This paper discusses the technical evolution of the Virtual Cluster product, and gives an overview of the science enabled. We discuss the challenges in supporting an ecosystem of these virtual clusters, and in supporting research on cloud resources in general.

[1]  Felix Bachmann,et al.  What is campus bridging and what is XSEDE doing about it? , 2012, XSEDE '12.

[2]  Srinath Perera,et al.  Apache airavata: a framework for distributed applications and computational workflows , 2011, GCE '11.

[3]  Nancy Wilkins-Diehr,et al.  XSEDE: Accelerating Scientific Discovery , 2014, Computing in Science & Engineering.

[4]  Nicholas T. Carnevale,et al.  Introducing The Neuroscience Gateway , 2013, IWSG.

[5]  Victor Hazlewood,et al.  Methods For Creating XSEDE Compatible Clusters , 2014, XSEDE '14.

[6]  Andy B. Yoo,et al.  Approved for Public Release; Further Dissemination Unlimited X-ray Pulse Compression Using Strained Crystals X-ray Pulse Compression Using Strained Crystals , 2002 .

[7]  Craig A. Stewart.  Big Data, Big Red II, Data Capacitor II, Wrangler, Jetstream, and Globus Online: Jetstream – a national science & engineering cloud , 2015 .

[8]  Evan F. Bollig,et al.  From Bare Metal to Virtual: Lessons Learned when a Supercomputing Institute Deploys its First Cloud , 2018, PEARC.

[9]  Ian T. Foster,et al.  Jetstream: a self-provisioned, scalable science and engineering cloud environment , 2015, XSEDE.

[10]  Andrew Lonie,et al.  CloudLaunch: Discover and Deploy Cloud Applications , 2018, Future Gener. Comput. Syst..

[11]  Paul Rad,et al.  Chameleon: A Scalable Production Testbed for Computer Science Research , 2019, Contemporary High Performance Computing.

[12]  David J. Scott,et al.  UltraScan - A Comprehensive Data Analysis Software Package for Analytical Ultracentrifugation Experiments , 2005 .

[13]  Kent Milfeld,et al.  From Proposal to Production: Lessons Learned Developing the Computational Chemistry Grid Cyberinfrastructure , 2006, Journal of Grid Computing.

[14]  Eric Coulter,et al.  Implementation of Simple XSEDE-Like Clusters: Science Enabled and Lessons Learned , 2016, XSEDE.

[15]  Marlon E. Pierce,et al.  Using the Jetstream Research Cloud to Provide Science Gateway Resources , 2017, 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID).