Ensuring data integrity in storage: techniques and applications

Data integrity is a fundamental aspect of storage security and reliability. With the advent of network storage and new technology trends that result in new failure modes for storage, interesting challenges arise in ensuring data integrity. In this paper, we discuss the causes of integrity violations in storage and present a survey of integrity assurance techniques that exist today. We describe several interesting applications of storage integrity checking, apart from security, and discuss the implementation issues associated with techniques. Based on our analysis, we discuss the choices and trade-offs associated with each mechanism. We then identify and formalize a new class of integrity assurance techniques that involve logical redundancy. We describe how logical redundancy can be used in today's systems to perform efficient and seamless integrity assurance.

[1]  Richard W. Hamming,et al.  Error detecting and error correcting codes , 1950 .

[2]  David Mazières,et al.  Fast and secure distributed read-only file system , 2000, TOCS.

[3]  Ronald L. Rivest,et al.  The MD5 Message-Digest Algorithm , 1992, RFC.

[4]  GhemawatSanjay,et al.  The Google file system , 2003 .

[5]  Hugo Krawczyk,et al.  HMAC: Keyed-Hashing for Message Authentication , 1997, RFC.

[6]  Darrell D. E. Long,et al.  Strong Security for Network-Attached Storage , 2002, FAST.

[7]  Erez Zadok,et al.  I3FS: An In-Kernel Integrity Checker and Intrusion Detection File System , 2004, LISA.

[8]  Paul Mackerras,et al.  The rsync algorithm , 1996 .

[9]  Erez Zadok,et al.  Proceedings of the General Track: 2003 Usenix Annual Technical Conference Ncryptfs: a Secure and Convenient Cryptographic File System , 2022 .

[10]  Yogen K. Dalal,et al.  Pilot: an operating system for a personal computer , 1980, CACM.

[11]  Erez Zadok,et al.  Fast Indexing: Support for Size-Changing Algorithms in Stackable File Systems , 2001, USENIX Annual Technical Conference, General Track.

[12]  Windsor W. Hsu,et al.  Fossilization: A process for establishing truly trustworthy records , 2004 .

[13]  Matt Blaze,et al.  A cryptographic file system for UNIX , 1993, CCS '93.

[14]  Andrea C. Arpaci-Dusseau,et al.  IRON file systems , 2005, SOSP '05.

[15]  Ernst W. Biersack,et al.  Performance evaluation of Forward Error Correction in ATM networks , 1992, SIGCOMM '92.

[16]  Marc Unangst,et al.  NASD Scalable Storage Systems , 1999 .

[17]  Craig A. N. Soules,et al.  Self-securing storage: protecting data in compromised systems , 2000, Foundations of Intrusion Tolerant Systems, 2003 [Organically Assured and Survivable Information Systems].

[18]  W. W. Peterson,et al.  Error-Correcting Codes. , 1962 .

[19]  Margo I. Seltzer,et al.  Unifying File System Protection , 2001, USENIX Annual Technical Conference, General Track.

[20]  Randy H. Katz,et al.  A case for redundant arrays of inexpensive disks (RAID) , 1988, SIGMOD '88.

[21]  Margo I. Seltzer,et al.  A New Hashing Package for UNIX , 1991, USENIX Winter.

[22]  Andrea C. Arpaci-Dusseau,et al.  Semantically-Smart Disk Systems , 2003, FAST.

[23]  R. J. McEliece,et al.  On sharing secrets and Reed-Solomon codes , 1981, CACM.

[24]  Erez Zadok,et al.  Cryptographic File Systems Performance: What You Don’t Know Can Hurt You , 2003, Second IEEE International Security in Storage Workshop.

[25]  Divesh Srivastava,et al.  Fault Tolerance Issues in Data Declustering for Parallel Database Systems. , 1994 .

[26]  Wesley D. Craig,et al.  Radmind: The Integration of Filesystem Integrity Checking with Filesystem Management , 2003, LISA.

[27]  Sean Quinlan,et al.  Venti: A New Approach to Archival Storage , 2002, FAST.

[28]  Kanishk Jain.  Object-based Storage , 2022 .

[29]  F. Lemmermeyer.  Error-correcting Codes , 2005 .

[30]  Sivan Toledo,et al.  A Transactional Flash File System for Microcontrollers , 2005, USENIX Annual Technical Conference, General Track.

[31]  Eugene H. Spafford,et al.  The design and implementation of tripwire: a file system integrity checker , 1994, CCS '94.

[32]  Robert B. Hagmann,et al.  Reimplementing the Cedar file system using logging and group commit , 1987, SOSP '87.

[33]  David D. Redell,et al.  Pilot: An operating system for a personal computer (Summary) , 1979, SOSP '79.

[34]  Erez Zadok,et al.  Enhancing File System Integrity Through Checksums , 2004 .

[35]  Eugene H. Spafford,et al.  Experiences with Tripwire: Using Integrity Checkers for Intrusion Detection , 1994 .

[36]  Lisa Spainhower,et al.  Commercial fault tolerance: a tale of two systems , 2004, IEEE Transactions on Dependable and Secure Computing.