Finding semantic bugs in file systems with an extensible fuzzing framework

File systems are too large to be bug free. Although handwritten test suites have been widely used to stress file systems, they can hardly keep up with the rapid increase in file system size and complexity, leading to new bugs being introduced and reported regularly. These bugs come in various flavors: simple buffer overflows to sophisticated semantic bugs. Although bug-specific checkers exist, they generally lack a way to explore file system states thoroughly. More importantly, no turnkey solution exists that unifies the checking effort of various aspects of a file system under one umbrella. In this paper, we highlight the potential of applying fuzzing to find not just memory errors but, in theory, any type of file system bugs with an extensible fuzzing framework: Hydra. Hydra provides building blocks for file system fuzzing, including input mutators, feedback engines, a libOS-based executor, and a bug reproducer with test case minimization. As a result, developers only need to focus on building the core logic for finding bugs of their own interests. We showcase the effectiveness of Hydra with four checkers that hunt crash inconsistency, POSIX violations, logic assertion failures, and memory errors. So far, Hydra has discovered 91 new bugs in Linux file systems, including one in a verified file system (FSCQ), as well as four POSIX violations.

[1]  Mathias Payer,et al.  T-Fuzz: Fuzzing by Program Transformation , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[2]  Xi Wang,et al.  Specifying and Checking File System Crash-Consistency Models , 2016, ASPLOS.

[3]  Abhik Roychoudhury,et al.  Directed Greybox Fuzzing , 2017, CCS.

[4]  Adam Chlipala,et al.  Proving confidentiality in a file system using DiskSec , 2018, OSDI.

[5]  Angela Demke Brown,et al.  Recon: Verifying file system consistency at runtime , 2012, TOS.

[6]  Nicolas Christin,et al.  Push-Button Verification of File Systems via Crash Refinement , 2016, USENIX Annual Technical Conference.

[7]  Chao Zhang,et al.  CollAFL: Path Sensitive Fuzzing , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[8]  Sebastian Schinzel,et al.  kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels , 2017, USENIX Security Symposium.

[9]  Junfeng Yang,et al.  Automatically generating malicious disks using symbolic execution , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[10]  Changwoo Min,et al.  Scalability in the Clouds!: A Myth or Reality? , 2015, APSys.

[11]  Peter F. Corbett,et al.  Row-Diagonal Parity for Double Disk Failure Correction (Awarded Best Paper!) , 2004, USENIX Conference on File and Storage Technologies.

[12]  Lisa Spainhower,et al.  Commercial fault tolerance: a tale of two systems , 2004, IEEE Transactions on Dependable and Secure Computing.

[13]  Insik Shin,et al.  Razzer: Finding Kernel Race Bugs through Fuzzing , 2019, 2019 IEEE Symposium on Security and Privacy (SP).

[14]  Josef Bacik,et al.  BTRFS: The Linux B-Tree Filesystem , 2013, TOS.

[15]  Taesoo Kim,et al.  Fuzzing File Systems via Two-Dimensional Input Space Exploration , 2019, 2019 IEEE Symposium on Security and Privacy (SP).

[16]  Junfeng Yang,et al.  Reducing crash recoverability to reachability , 2016, POPL.

[17]  Alex Groce,et al.  Randomized Differential Testing as a Prelude to Formal Verification , 2007, 29th International Conference on Software Engineering (ICSE'07).

[18]  Abhik Roychoudhury,et al.  Coverage-Based Greybox Fuzzing as Markov Chain , 2016, IEEE Transactions on Software Engineering.

[19]  Herbert Bos,et al.  VUzzer: Application-aware Evolutionary Fuzzing , 2017, NDSS.

[20]  Sang Kil Cha,et al.  IMF: Inferred Model-based Fuzzer , 2017, CCS.

[21]  Nicolae Tapus,et al.  LKL: The Linux kernel library , 2010, 9th RoEduNet IEEE International Conference.

[22]  Junfeng Yang,et al.  EXPLODE: a lightweight, general system for finding serious storage system errors , 2006, OSDI '06.

[23]  Andrea C. Arpaci-Dusseau,et al.  A Study of Linux File System Evolution , 2013, FAST.

[24]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2017, SOSP.

[25]  Adam Chlipala,et al.  Verifying a high-performance crash-safe file system using a tree specification , 2017, SOSP.

[26]  Changwoo Min,et al.  Cross-checking semantic correctness: the case of finding file system bugs , 2015, SOSP.

[27]  Tom Ridge,et al.  SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems , 2015, SOSP.

[28]  Wenke Lee,et al.  Unleashing Use-Before-Initialization Vulnerabilities in the Linux Kernel Using Targeted Stack Spraying , 2017, NDSS.

[29]  Rodrigo Rodrigues,et al.  SKI: Exposing Kernel Concurrency Bugs through Systematic Schedule Exploration , 2014, OSDI.

[30]  Suman Jana,et al.  MoonShine: Optimizing OS Fuzzer Seed Selection with Trace Distillation , 2018, USENIX Security Symposium.

[31]  Suparna Bhattacharya,et al.  Ext4: The Next Generation of Ext2/3 Filesystem , 2007, LSF.

[32]  Junfeng Yang,et al.  Using model checking to find serious file system errors , 2004, TOCS.

[33]  Pandian Raju,et al.  Finding Crash-Consistency Bugs with Bounded Black-Box Crash Testing , 2018, OSDI.

[34]  Andrea C. Arpaci-Dusseau,et al.  IRON file systems , 2005, SOSP '05.

[35]  Adam Chlipala,et al.  Using Crash Hoare logic for certifying the FSCQ file system , 2015, USENIX Annual Technical Conference.