File server scaling with network-attached secure disks

By providing direct data transfer between storage and client, network-attached storage devices have the potential to improve scalability for existing distributed file systems (by removing the server as a bottleneck) and bandwidth for new parallel and distributed file systems (through network striping and more efficient data paths). Together, these advantages influence a large enough fraction of the storage market to make commodity network-attached storage feasible. Realizing the technology's full potential requires careful consideration across a wide range of file system, networking and security issues. This paper contrasts two network-attached storage architectures---(1) Networked SCSI disks (NetSCSI) are network-attached storage devices with minimal changes from the familiar SCSI interface, while (2) Network-Attached Secure Disks (NASD) are drives that support independent client access to drive object services. To estimate the potential performance benefits of these architectures, we develop an analytic model and perform trace-driven replay experiments based on AFS and NFS traces. Our results suggest that NetSCSI can reduce file server load during a burst of NFS or AFS activity by about 30%. With the NASD architecture, server load (during burst activity) can be reduced by a factor of up to five for AFS and up to ten for NFS.

[1]  Peter Steenkiste.  A systematic approach to host interface design for high-speed networks , 1994, Computer.

[2]  Bennet S. Yee,et al.  Secure Coprocessors in Electronic Commerce Applications , 1995, USENIX Workshop on Electronic Commerce.

[3]  Howard Gobioff,et al.  Security for Network Attached Storage Devices (CMU-CS-97-185) , 1997 .

[4]  Anna R. Karlin,et al.  Implementing global memory management in a workstation cluster , 1995, SOSP.

[5]  Wilson C. Hsieh,et al.  The logical disk: a new approach to improving file systems , 1994, SOSP '93.

[6]  Greg Minshall,et al.  An Overview of the NetWare Operating System , 1994, USENIX Winter.

[7]  Jonathan M. Smith,et al.  Striping within the network subsystem , 1995, IEEE Netw..

[8]  Michelle Y. Kim,et al.  Synchronized Disk Interleaving , 1986, IEEE Transactions on Computers.

[9]  José Carlos Brustoloni,et al.  Effects of buffering semantics on I/O performance , 1996, OSDI '96.

[10]  John A. Kunze,et al.  A trace-driven analysis of the UNIX 4.2 BSD file system , 1985, SOSP '85.

[11]  Alan Eustace,et al.  ATOM - A System for Building Customized Program Analysis Tools , 1994, PLDI.

[12]  Mahadev Satyanarayanan,et al.  Scale and performance in a distributed file system , 1987, SOSP '87.

[13]  H. T. Kung,et al.  The design of nectar: a network backplane for heterogeneous multicomputers , 1989, ASPLOS III.

[14]  Larry L. Peterson,et al.  Fbufs: a high-bandwidth cross-domain transfer facility , 1994, SOSP '93.

[15]  David Clark,et al.  An analysis of TCP processing overhead , 1989 .

[16]  J. Howard Et El,et al.  Scale and performance in a distributed file system , 1988 .

[17]  Jim Zelenka,et al.  Informed prefetching and caching , 1995, SOSP.

[18]  Darrell D. E. Long,et al.  Swift/RAID: A Distributed RAID , 1994 .

[19]  Stephen W. Miller,et al.  A Reference Model for Mass Storage Systems , 1988, Adv. Comput..

[20]  Steve H. Weingart.  Physical Security for the μABYSS System , 1987, 1987 IEEE Symposium on Security and Privacy.

[21]  Elizabeth Shriver,et al.  Attribute-managed storage , 1995 .

[22]  Steve R. White,et al.  ABYSS: ATrusted Architecture for Software Protection , 1987, 1987 IEEE Symposium on Security and Privacy.

[23]  Dan Walsh,et al.  Design and implementation of the Sun network filesystem , 1985, USENIX Conference Proceedings.

[24]  Garth A. Gibson,et al.  Understanding Customer Dissatisfaction with Underutilized Distributed File Servers (CMU-CS-96-158) , 1996 .

[25]  Seth Copen Goldstein,et al.  Active messages: a mechanism for integrating communication and computation , 1998, ISCA '98.

[26]  Randy H. Katz,et al.  An evaluation of redundant arrays of disks using an Amdahl 5890 , 1990, SIGMETRICS '90.

[27]  John H. Hartman,et al.  The Zebra striped network file system , 1995, TOCS.

[28]  John Kunze,et al.  A trace-driven analysis of the unix 4 , 1985, SOSP 1985.

[29]  Avraham Leff,et al.  LAN and I/O convergence: a survey of the issues , 1994, Computer.

[30]  Edward K. Lee.  Highly-available, scalable network storage , 1995, Digest of Papers. COMPCON'95. Technologies for the Information Superhighway.

[31]  Garth A. Gibson.  Redundant disk arrays: Reliable, parallel secondary storage. Ph.D. Thesis , 1990 .

[32]  H. T. Kung,et al.  The design of nectar: a network backplane for heterogeneous multicomputers , 1989, ASPLOS 1989.

[33]  Miron Livny,et al.  Multi-disk management algorithms , 1987, SIGMETRICS '87.

[34]  Srinivasan Seshan,et al.  RAID-II: a high-bandwidth network file server , 1994, ISCA '94.

[35]  Chris Ruemmler,et al.  Disk Shuffling , 1991 .

[36]  Brian N. Bershad,et al.  The interaction of architecture and operating system design , 1991, ASPLOS IV.

[37]  Mahadev Satyanarayanan,et al.  Scale and performance in a distributed file system , 1988, TOCS.

[38]  Carl Staelin,et al.  The HP AutoRAID hierarchical storage system , 1995, SOSP.

[39]  Anna R. Karlin,et al.  A study of integrated prefetching and caching strategies , 1995, SIGMETRICS '95/PERFORMANCE '95.

[40]  John K. Ousterhout,et al.  Why Aren't Operating Systems Getting Faster As Fast as Hardware? , 1990, USENIX Summer.

[41]  Howard Gobioff,et al.  Security for Network Attached Storage Devices , 1997 .

[42]  Michael Dahlin,et al.  Cooperative caching: using remote client memory to improve file system performance , 1994, OSDI '94.

[43]  F. Bitz,et al.  Host interface design for ATM LANs , 1991, [1991] Proceedings 16th Conference on Local Computer Networks.

[44]  Brian N. Bershad,et al.  Protocol service decomposition for high-performance networking , 1994, SOSP '93.

[45]  K OusterhoutJohn,et al.  Caching in the Sprite network file system , 1988 .

[46]  P. M. Chm.  An evaluation of redundant arrays of disks using an amdahl5890 , 1990, SIGMETRICS 1990.

[47]  Richard W. Watson,et al.  The parallel I/O architecture of the high-performance storage system (HPSS) , 1995, Proceedings of IEEE 14th Symposium on Mass Storage Systems.

[48]  Brian N. Bershad,et al.  The impact of operating system structure on memory system performance , 1994, SOSP '93.

[49]  Randy H. Katz,et al.  High-performance network and channel-based storage , 1992, Proc. IEEE.

[50]  James Lau,et al.  File System Design for an NFS File Server Appliance , 1994, USENIX Winter.

[51]  Randy H. Katz,et al.  A case for redundant arrays of inexpensive disks (RAID) , 1988, SIGMOD '88.

[52]  Charles L. Seitz,et al.  Myrinet: A Gigabit-per-Second Local Area Network , 1995, IEEE Micro.

[53]  Gregory G. Finn,et al.  Derived virtual devices: a secure distributed file system mechanism , 1996 .

[54]  Robert W. Horst.  TNet: A Reliable System Area Network , 1995, IEEE Micro.

[55]  John H. Hartman,et al.  Zebra: A Striped Network File System , 1992 .

[56]  Darrell D. E. Long,et al.  Swift/RAID: A Distributed RAID System , 1994, Comput. Syst..

[57]  Alan F. Benner.  Fibre Channel: Gigabit Communications and I/O for Computer Networks , 1995 .

[58]  Nancy P. Kronenberg,et al.  VAXcluster: a closely-coupled distributed system , 1986, TOCS.

[59]  Rodney Van Meter,et al.  A Brief Survey of Current Work on Network Attached Peripherals (Extended Abstract) , 1996, ACM SIGOPS Oper. Syst. Rev..

[60]  Yelena Yesha,et al.  An analytic model of hierarchical mass storage systems with network-attached storage devices , 1996, SIGMETRICS '96.

[61]  Peter Steenkiste,et al.  Routing high-bandwidth traffic in max-min fair share networks , 1996, SIGCOMM '96.

[62]  Raj Jain,et al.  A brief overview of ATM: protocol layers, LAN emulation, and traffic management , 1995, CCRV.

[63]  Michael N. Nelson,et al.  Caching in the Sprite network file system , 1988, TOCS.

[64]  Quinn Jacobson,et al.  Destage algorithms for disk arrays with non-volatile caches , 1995, Proceedings 22nd Annual International Symposium on Computer Architecture.

[65]  Michael Dahlin,et al.  A quantitative analysis of cache policies for scalable network file systems , 1994, SIGMETRICS.

[66]  E. Grochowski,et al.  Future trends in hard disk drives , 1996 .

[67]  Roger M. Needham,et al.  A Universal File Server , 1980, IEEE Transactions on Software Engineering.

[68]  Mary Baker,et al.  Measurements of a distributed file system , 1991, SOSP '91.