Managing Very-Large Distributed Datasets

In this paper, we introduce a system for handling very large datasets, which need to be stored across multiple computing sites. Data distribution introduces complex management issues, particularly as computing sites may make use of different storage systems with different internal organizations. The motivation for our work is the ATLAS Experiment for the Large Hadron Collider (LHC) at CERN, where the authors are involved in developing the data management middleware. This middleware, called DQ2, is charged with shipping petabytes of data every month to research centers and universities worldwide and has achieved aggregate throughputs in excess of 1.5 Gbytes/sec over the wide-area network. We describe DQ2's design and implementation, which builds upon previous work on distributed file systems, peer-to-peer systems and Data Grids. We discuss its fault tolerance and scalability properties and briefly describe results from its daily usage for the ATLAS Experiment.

[1]  Roy Fielding,et al.  Architectural Styles and the Design of Network-based Software Architectures"; Doctoral dissertation , 2000 .

[2]  Tim Moors,et al.  Survey of Research towards Robust Peer-to-Peer Networks: Search Methods , 2007, RFC.

[3]  Edith Cohen,et al.  Replication strategies in unstructured peer-to-peer networks , 2002, SIGCOMM.

[4]  Ian T. Foster,et al.  The data grid: Towards an architecture for the distributed management and analysis of large scientific datasets , 2000, J. Netw. Comput. Appl..

[5]  Heinz Stockinger,et al.  Grid Data Management Pilot (GDMP): A Tool for Wide Area Replication , 2001 .

[6]  E. Deelman,et al.  Data replication strategies in grid environments , 2002, Fifth International Conference on Algorithms and Architectures for Parallel Processing, 2002. Proceedings..

[7]  Kurt Stockinger,et al.  Simulation of Dynamic Grid Replication Strategies in OptorSim , 2002, GRID.

[8]  Manish Parashar,et al.  Grid Computing — GRID 2002 , 2002, Lecture Notes in Computer Science.

[9]  Andrew J. Hutton,et al.  Lustre: Building a File System for 1,000-node Clusters , 2003 .

[10]  W. Allcock,et al.  GridFTP protocol specification , 2002 .

[11]  Peter Z. Kunszt,et al.  Giggle: A Framework for Constructing Scalable Replica Location Services , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[12]  A. Frohner,et al.  Data storage, access and catalogs in gLite , 2005, 2005 IEEE International Symposium on Mass Storage Systems and Technology.

[13]  Edith Cohen,et al.  Search and replication in unstructured peer-to-peer networks , 2002, ICS '02.

[14]  Dan Walsh,et al.  Design and implementation of the Sun network filesystem , 1985, USENIX Conference Proceedings.

[15]  Edith Cohen,et al.  Search and replication in unstructured peer-to-peer networks , 2002 .

[16]  Frank B. Schmuck,et al.  GPFS: A Shared-Disk File System for Large Computing Clusters , 2002, FAST.

[17]  Mario Lassnig,et al.  Monitoring the atlas distributed data management system , 2008 .

[18]  Arie Shoshani,et al.  Storage resource managers: Middleware components for gridstorage , 2005 .

[19]  Howard Gobioff,et al.  The Google file system , 2003, SOSP '03.

[20]  Ian T. Foster,et al.  GASS: a data movement and access service for wide area computing systems , 1999, IOPADS '99.

[21]  Phil Andrews,et al.  Massive High-Performance Global File Systems for Grid computing , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[22]  Ian T. Foster,et al.  A security architecture for computational grids , 1998, CCS '98.

[23]  Reagan Moore,et al.  The SDSC storage resource broker , 2010, CASCON.

[24]  J. Howard Et El,et al.  Scale and performance in a distributed file system , 1988 .

[25]  Mahadev Satyanarayanan,et al.  Coda: A Highly Available File System for a Distributed Workstation Environment , 1990, IEEE Trans. Computers.