Cybercosm: New Foundations for a Converged Science Data Ecosystem

Scientific communities naturally tend to organize around data ecosystems created by the combination of their observational devices, their data repositories, and the workflows essential to carry their research from observation to discovery. However, these legacy data ecosystems are now breaking down under the pressure of the exponential growth in the volume and velocity of these workflows, which are further complicated by the need to integrate the highly data intensive methods of the Artificial Intelligence (AI) revolution. Enabling ground breaking science that makes full use of this new, data saturated research environment will require distributed systems that support dramatically improved resource sharing, workflow portability and composability, and data ecosystem convergence. The Cybercosm vision presented in this white paper describes a radically different approach to the architecture of distributed systems for data-intensive science and its application workflows. As opposed to traditional models that restrict interoperability by hiving off storage, networking, and computing resources in separate technology silos, Cybercosm defines a minimally sufficient hypervisor as a spanning layer for its “data plane” that virtualizes and converges the local resources of the system’s nodes in a fully interoperable manner. By building on a common, universal interface into which the problems that infect today’s data-intensive workflows can be decomposed and attacked, Cybercosm aims to support scalable, portable and composable workflows that span and merge the distributed data ecosystems that characterize leading edge research communities today.

[1]  Francine Berman,et al.  Logistical quality of service in NetSolve , 1999, Comput. Commun..

[2]  David G. Messerschmitt,et al.  Software Ecosystem: Understanding an Indispensable Technology and Industry , 2003 .

[3]  Scott Klasky,et al.  High performance threaded data streaming for large scale simulations , 2004, Fifth IEEE/ACM International Workshop on Grid Computing.

[4]  D. Martin Swany,et al.  Multi-Layer Stream Orchestration with Flange , 2019, 2019 IEEE International Conference on Fog Computing (ICFC).

[5]  Piotr Luszczek,et al.  Interoperable Convergence of Storage, Networking and Computation , 2017, Lecture Notes in Networks and Systems.

[6]  Andrew G. Salinger,et al.  Albany/FELIX : a parallel, scalable and robust, finite element, first-order Stokes approximation ice sheet solver built for advanced analysis , 2014 .

[7]  Alessandro Bassi,et al.  Mobile management of network files , 2001, Proceedings Third Annual International Workshop on Active Middleware Services.

[8]  Erik Elmroth,et al.  Inter-Disciplinary Research Challenges in Computer Systems for the 2020s , 2018 .

[9]  David Wetherall,et al.  Towards an active network architecture , 1996, CCRV.

[10]  Ian T. Foster,et al.  The anatomy of the grid: enabling scalable virtual organizations , 2001, Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid.

[11]  Micah Beck,et al.  An end-to-end approach to globally scalable programmable networking , 2003, FDNA '03.

[12]  Chip Elliott,et al.  GENI - global environment for network innovations , 2008, LCN.

[13]  John Seely Brown,et al.  Shaping Strategies for the IoT , 2017, Computer.

[14]  The Ligo Scientific Collaboration,et al.  Observation of Gravitational Waves from a Binary Black Hole Merger , 2016, 1602.03837.

[15]  Larry Smarr,et al.  The Pacific Research Platform: Making High-Speed Networking a Reality for the Scientist , 2018, PEARC.

[17]  Micah Beck,et al.  On the hourglass model , 2016, Commun. ACM.

[18]  Micah Beck,et al.  Harnessing the Computing Continuum for Programming Our World , 2020, Fog Computing.

[19]  Thomas Hérault,et al.  PaRSEC: Exploiting Heterogeneity to Enhance Scalability , 2013, Computing in Science & Engineering.

[20]  Jaroslava Schovancova,et al.  Architecture and prototype of a WLCG data lake for HL-LHC , 2019, EPJ Web of Conferences.

[21]  Larry L. Peterson,et al.  Experiences building PlanetLab , 2006, OSDI '06.