Customizing the swarm storage system using agents

Swarm is a scalable, modular storage system that uses agents to customize low-level storage functions to meet the needs of high-level services. Agents influence low-level storage functions such as data layout, metadata management, and crash recovery. An agent is a program that is attached to data in the storage system and invoked when events occur during the data's lifetime. For example, before Swarm writes data to disk, agents attached to the data are invoked to determine a layout policy. Agents are typically persistent, remaining attached to the data they manage until the data are deleted; this allows agents to continue to affect how the data are handled long after the application or storage service that created the data has terminated. In this paper, we present Swarm's agent architecture, describe the types of agents that Swarm supports and the infrastructure used to support them, and discuss their performance overhead and security implications. We describe how several storage services and applications use agents, and the benefits they derive from doing so. Copyright © 2005 John Wiley & Sons, Ltd.

[1]  Gregory R. Ganger,et al.  Dynamic Function Placement for Data-Intensive Cluster Computing , 2000, USENIX Annual Technical Conference, General Track.

[2]  Mendel Rosenblum,et al.  The design and implementation of a log-structured file system , 1991, SOSP '91.

[3]  Peter F. Corbett,et al.  Proposal for a Common Parallel File System Programming Interface 1.0 , 1996, WWW 1996.

[4]  George C. Necula,et al.  Safe kernel extensions without run-time checking , 1996, OSDI '96.

[5]  Robert Grimm,et al.  Atomic recovery units: failure atomicity for logical disks , 1996, Proceedings of 16th International Conference on Distributed Computing Systems.

[6]  Wilson C. Hsieh,et al.  The logical disk: a new approach to improving file systems , 1994, SOSP '93.

[7]  John H. Hartman,et al.  The Swarm scalable storage system , 1999, Proceedings. 19th IEEE International Conference on Distributed Computing Systems (Cat. No.99CB37003).

[8]  Andrea C. Arpaci-Dusseau,et al.  Evolving RPC for active storage , 2002, ASPLOS X.

[9]  Larry L. Peterson,et al.  The x-Kernel: An Architecture for Implementing Network Protocols , 1991, IEEE Trans. Software Eng..

[10]  Robert Grimm,et al.  Application performance and flexibility on exokernel systems , 1997, SOSP.

[11]  Jim Zelenka,et al.  File server scaling with network-attached secure disks , 1997, SIGMETRICS '97.

[12]  Garth A. Gibson,et al.  Active Disks: Remote Execution for Network-Attached Storage (CMU-CS-97-198) , 1997 .

[13]  Robert S. Fabry,et al.  A fast file system for UNIX , 1984, TOCS.

[14]  John H. Hartman,et al.  The Zebra striped network file system , 1995, TOCS.

[15]  Margo I. Seltzer,et al.  Dealing with disaster: surviving misbehaved kernel extensions , 1996, OSDI '96.

[16]  David L. Black,et al.  Machine-independent virtual memory management for paged uniprocessor and multiprocessor architectures , 1987, ASPLOS 1987.

[17]  Robert Wahbe,et al.  Efficient software-based fault isolation , 1994, SOSP '93.