Tertiary Disk: Large Scale Distributed Storage

In the past 5 years, disk costs have been falling at a factor of 2 per year. Today, terabyte capacity disk storage systems are feasible. Given the rapidly increasing areal density and disk transfer rates, these systems will have significant cost/performance advantages over tape libraries of similar capacity. If commodity hardware is used, large disk systems can avoid the high cost of custom designed disk arrays, as well as their limitations on scalability. This paper presents Tertiary Disk, a 3TB disk storage system built from commodity hardware. Tertiary Disk uses PCs and switched networks to connect 370 8GB disks. We show that even though commodity hardware is used, the overall system can be more reliable than a single disk. A cost analysis of our prototype shows that the additional infrastructure needed to create a terabyte scale storage system is a fraction of the cost of the underlying disks. In comparison, the costs of large disk arrays are many times the cost of the underlying disks. We also present performance measurements from our prototype, and show that the PC architecture is a good match for hosting a large number of disks. Overall, we show that storage system designs like Tertiary Disk have cost/performance and reliability advantages over most choices available today for terabyte scale storage.