HTTP redirection for replica catalogue lookups in data grids

Data distribution and replication in distributed systems require special purpose middleware tools for accessing replicated data. Data Grids, special forms of systems distributed over wide-area networks, need to handle data management issues like distribution and replication of large amounts of data in the Tera- and Petabyte scale. Replica catalogues are used for cataloguing and locating replicated files in distributed sites all around the globe. We present a novel and administratively scalable approach for distributing a replica catalogue and resolving file location information by using HTTP redirection. HTTP redirection servers managing local file catalogues allow for greater flexibility and local file management autonomy whereas a global replica catalogue provides the necessary mapping of logical files to individual sites. By distributing the catalogues a site can autonomously move files for load balancing within a site without notifying a global replica catalogue. Our approach scales well in terms of catalogue administration to a large number of sites and file entries and thus establishes a powerful middleware service. We present the design and implementation of our catalogue redirection servers and report on promising experimental results.