Verifying distributed erasure-coded data

Erasure coding can reduce the space and band width overheads of redundancy in fault-tolerant data storage and delivery systems. But it introduces the fundamental difficulty of ensuring that all erasure-coded fragments correspond to the same block of data. Without such assurance, a different block may be reconstructed from different subsets of fragments. This paper develops a technique for providing this assurance without the bandwidth and computational overheads associated with current approaches. The core idea is to distribute with each fragment what we call homomorphic fingerprints. These fingerprints preserve the structure of the erasure code and allow each fragment to be independently verified as corresponding to a specific block. We demonstrate homomorphic fingerprinting functions that are secure, efficient, and compact.

[1]  Dennis Travis On Irreducible Polynomials in Galois Fields , 1963 .

[2]  Larry Carter,et al.  Universal classes of hash functions (Extended Abstract) , 1977, STOC '77.

[3]  Baruch Awerbuch,et al.  Verifiable secret sharing and achieving simultaneity in the presence of faults , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[4]  Paul Feldman,et al.  A practical scheme for non-interactive verifiable secret sharing , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[5]  Michael O. Rabin,et al.  Efficient dispersal of information for security, load balancing, and fault tolerance , 1989, JACM.

[6]  Li Gong Securely replicating authentication services , 1989, [1989] Proceedings. The 9th International Conference on Distributed Computing Systems.

[7]  Torben P. Pedersen Distributed Provers with Applications to Undeniable Signatures , 1991, EUROCRYPT.

[8]  A. Broder Some applications of Rabin’s fingerprinting method , 1993 .

[9]  Hugo Krawczyk Distributed fingerprints and secure information dispersal , 1993, PODC '93.

[10]  Mihir Bellare,et al.  Random oracles are practical: a paradigm for designing efficient protocols , 1993, CCS '93.

[11]  Mihir Bellare,et al.  Incremental Cryptography: The Case of Hashing and Signing , 1994, CRYPTO.

[12]  Hugo Krawczyk,et al.  LFSR-based Hashing and Authentication , 1994, CRYPTO.

[13]  Markus Stadler,et al.  Publicly Verifiable Secret Sharing , 1996, EUROCRYPT.

[14]  Victor Shoup,et al.  On Fast and Provably Secure Message Authentication Based on Universal Hashing , 1996, CRYPTO.

[15]  Bart Preneel,et al.  Software Performance of Universal Hash Functions , 1999, EUROCRYPT.

[16]  Moni Naor,et al.  Distributed Pseudo-random Functions and KDCs , 1999, EUROCRYPT.

[17]  Ben Y. Zhao,et al.  OceanStore: an architecture for global-scale persistent storage , 2000, SIGP.

[18]  P. Maymounkov Online codes , 2002 .

[19]  Miguel Castro,et al.  SplitStream: high-bandwidth multicast in cooperative environments , 2003, SOSP '03.

[20]  Michael K. Reiter,et al.  Efficient Byzantine-tolerant erasure-coded storage , 2004, International Conference on Dependable Systems and Networks, 2004.

[21]  Thomas Shrimpton,et al.  Cryptographic Hash-Function Basics: Definitions, Implications, and Separations for Preimage Resistance, Second-Preimage Resistance, and Collision Resistance , 2004, FSE.

[22]  Arif Merchant,et al.  FAB: building distributed enterprise disk arrays from commodity components , 2004, ASPLOS XI.

[23]  David Mazières,et al.  On-the-fly verification of rateless erasure codes for efficient content distribution , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[24]  Michael K. Reiter,et al.  The Safety and Liveness Properties of a Protocol Family for Versatile Survivable Storage Infrastructures (CMU-PDL-03-105) , 2004 .

[25]  Thomas J. E. Schwarz Verification of Parity Data in Large Scale Storage Systems , 2004, PDPTA.

[26]  Kurt Mehlhorn,et al.  Randomized and deterministic simulations of PRAMs by parallel machines with restricted granularity of parallel memories , 1984, Acta Informatica.

[27]  Marcos K. Aguilera,et al.  Using erasure codes efficiently for storage in a distributed system , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[28]  Stefano Tessaro,et al.  Optimal Resilience for Erasure-Coded Byzantine Distributed Storage , 2005, DISC.

[29]  Stefano Tessaro,et al.  Asynchronous Verifiable Information Dispersal , 2005, DISC.