A Comprehensive and Cross-Platform Test Suite for Memory Safety - Towards an Open Framework for Testing Processor Hardware Supported Security Extensions

Memory safety remains a critical and widely violated property in reality. Numerous defense techniques have been proposed and developed but most of them are not applied or enabled by default in production-ready environment due to their substantial running cost. The situation might change in the near future because the hardware supported defenses against these attacks are finally beginning to be adopted by commercial processors, operating systems and compilers. We then face a question as there is currently no suitable test suite to measure the memory safety extensions supported on different processors. In fact, the issue is not constrained only for memory safety but all aspect of processor security. All of the existing test suites related to processor security lack some of the key properties, such as comprehensiveness, distinguishability and portability. As an initial step, we propose an expandable test framework for measuring the processor security and open source a memory safety test suite utilizing this framework. The framework is deliberately designed to be flexible so it can be gradually extended to all types of hardware supported security extensions in processors. The initial test suite for memory safety currently contains 160 test cases covering spatial and temporal safety of memory, memory access control, pointer integrity and control-flow integrity. Each type of vulnerabilities and their related defenses have been individually evaluated by one or more test cases. The test suite has been ported to three different instruction set architectures (ISAs) and experimented on six different platforms. We have also utilized the test suite to explore the security benefits of applying different sets of compiler flags available on the latest GNU GCC and LLVM compilers.

[1]  Sam Ainsworth,et al.  MuonTrap: Preventing Cross-Domain Spectre-Like Attacks by Capturing Speculative State , 2019, 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA).

[2]  Milo M. K. Martin,et al.  Watchdog: Hardware for safe and secure manual memory management and full memory safety , 2012, 2012 39th Annual International Symposium on Computer Architecture (ISCA).

[3]  A. One,et al.  Smashing The Stack For Fun And Profit , 1996 .

[4]  Fan Long,et al.  Control Jujutsu: On the Weaknesses of Fine-Grained Control Flow Integrity , 2015, CCS.

[5]  John Johansen,et al.  PointGuard™: Protecting Pointers from Buffer Overflow Vulnerabilities , 2003, USENIX Security Symposium.

[6]  Gururaj Saileshwar,et al.  CleanupSpec: An "Undo" Approach to Safe Speculation , 2019, MICRO.

[7]  Zhenkai Liang,et al.  Data-Oriented Programming: On the Expressiveness of Non-control Data Attacks , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[8]  Guru Venkataramani,et al.  MemTracker: Efficient and Programmable Support for Memory Access Monitoring and Debugging , 2007, 2007 IEEE 13th International Symposium on High Performance Computer Architecture.

[9]  Richard P. Lippmann,et al.  Using a Diagnostic Corpus of C Programs to Evaluate Buffer Overflow Detection by Static Analysis Tools , 2005 .

[10]  Periklis Akritidis,et al.  Cling: A Memory Allocator to Mitigate Dangling Pointers , 2010, USENIX Security Symposium.

[11]  Michael Hamburg,et al.  Spectre Attacks: Exploiting Speculative Execution , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[12]  M. Zhivich.  Dynamic Buffer Overflow Detection , 2005 .

[13]  Emery D. Berger,et al.  DieHarder: securing the heap , 2010, CCS '10.

[14]  David Chisnall,et al.  Pointer Tagging for Memory Safety , 2019 .

[15]  Satish Narayanasamy,et al.  DOLMA: Securing Speculation with the Principle of Transient Non-Observability , 2021, USENIX Security Symposium.

[16]  Xuxian Jiang,et al.  On the Expressiveness of Return-into-libc Attacks , 2011, RAID.

[17]  Chester Rebeiro,et al.  SHAKTI-MS: a RISC-V processor for memory safety in C , 2019, LCTES.

[18]  Ahmad-Reza Sadeghi,et al.  Just-In-Time Code Reuse: On the Effectiveness of Fine-Grained Address Space Layout Randomization , 2013, 2013 IEEE Symposium on Security and Privacy.

[19]  Úlfar Erlingsson,et al.  Enforcing Forward-Edge Control-Flow Integrity in GCC & LLVM , 2014, USENIX Security Symposium.

[20]  William R. Harris,et al.  Efficient Protection of Path-Sensitive Control Security , 2017, USENIX Security Symposium.

[21]  Josep Torrellas,et al.  Secure hierarchy-aware cache replacement policy (SHARP): Defending against cache-based side channel attacks , 2017, 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA).

[22]  Mihai Budiu,et al.  Control-flow integrity principles, implementations, and applications , 2009, TSEC.

[23]  Frank Piessens,et al.  Fallout: Leaking Data on Meltdown-resistant CPUs , 2019, CCS.

[24]  Peter G. Neumann,et al.  The CHERI capability model: Revisiting RISC in an age of risk , 2014, 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA).

[25]  Herbert Bos,et al.  Out of Control: Overcoming Control-Flow Integrity , 2014, 2014 IEEE Symposium on Security and Privacy.

[26]  Dan Boneh,et al.  CCFI: Cryptographically Enforced Control Flow Integrity , 2015, CCS.

[27]  George Candea,et al.  Code-pointer integrity , 2014, OSDI.

[28]  Simha Sethumadhavan,et al.  Reviving instruction set randomization , 2017, 2017 IEEE International Symposium on Hardware Oriented Security and Trust (HOST).

[29]  Mathias Payer,et al.  SoK: Shining Light on Shadow Stacks , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[30]  Mario Werner,et al.  ScatterCache: Thwarting Cache Attacks via Cache Set Randomization , 2019, USENIX Security Symposium.

[31]  Ahmad-Reza Sadeghi,et al.  Counterfeit Object-oriented Programming: On the Difficulty of Preventing Code Reuse Attacks in C++ Applications , 2015, 2015 IEEE Symposium on Security and Privacy.

[32]  Jonathan D. Pincus,et al.  Beyond stack smashing: recent advances in exploiting buffer overruns , 2004, IEEE Security & Privacy Magazine.

[33]  Mingzhe Wang,et al.  Finding Cracks in Shields: On the Security of Control Flow Integrity Mechanisms , 2020, CCS.

[34]  Reinhold Weicker,et al.  Dhrystone: a synthetic systems programming benchmark , 1984, CACM.

[35]  Paul E. Black,et al.  Juliet 1.1 C/C++ and Java Test Suite , 2012, Computer.

[36]  Todd M. Austin,et al.  Smokestack: Thwarting DOP Attacks with Runtime Stack Layout Randomization , 2019, 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO).

[37]  André DeHon,et al.  Protecting the Stack with Metadata Policies and Tagged Hardware , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[38]  Jonathan M. Smith,et al.  Low-fat pointers: compact encoding and efficient gate-level implementation of fat pointers for spatial safety and capability-based security , 2013, CCS.

[39]  Ingrid Verbauwhede,et al.  Systematic Analysis of Randomization-based Protected Cache Architectures , 2021, 2021 IEEE Symposium on Security and Privacy (SP).

[40]  Olatunji Ruwase,et al.  A Practical Dynamic Buffer Overflow Detector , 2004, NDSS.

[41]  Wouter Joosen,et al.  RIPE: runtime intrusion prevention evaluator , 2011, ACSAC '11.

[42]  David H. Ackley,et al.  Building diverse computer systems , 1997, Proceedings. The Sixth Workshop on Hot Topics in Operating Systems (Cat. No.97TB100133).

[43]  Zhenkai Liang,et al.  Jump-oriented programming: a new class of code-reuse attack , 2011, ASIACCS '11.

[44]  Ahmad-Reza Sadeghi,et al.  Stitching the Gadgets: On the Ineffectiveness of Coarse-Grained Control-Flow Integrity Protection , 2014, USENIX Security Symposium.

[45]  Yves Younan,et al.  FreeSentry: protecting against use-after-free vulnerabilities due to dangling pointers , 2015, NDSS.

[46]  John L. Henning.  SPEC CPU2006 benchmark descriptions , 2006, CARN.

[47]  Milo M. K. Martin,et al.  WatchdogLite: Hardware-Accelerated Compiler-Based Pointer Checking , 2014, CGO '14.

[48]  Srivaths Ravi,et al.  Architectural support for safe software execution on embedded processors , 2006, Proceedings of the 4th International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS '06).

[49]  Sorin Lerner,et al.  SafeDispatch: Securing C++ Virtual Calls from Memory Corruption Attacks , 2014, NDSS.

[50]  Jakub Szefer,et al.  A Benchmark Suite for Evaluating Caches' Vulnerability to Timing Attacks , 2019, ASPLOS.

[51]  Yunheung Paek,et al.  HDFI: Hardware-Assisted Data-Flow Isolation , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[52]  Eugene H. Spafford,et al.  The internet worm program: an analysis , 1989, CCRV.

[53]  Thomas F. Wenisch,et al.  Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution , 2018, USENIX Security Symposium.

[54]  Angelos D. Keromytis,et al.  Kernel Protection Against Just-In-Time Code Reuse , 2019, ACM Trans. Priv. Secur..

[55]  Tao Li,et al.  BASS: a benchmark suite for evaluating architectural security systems , 2006, CARN.

[56]  Cesar Pereida García,et al.  Port Contention for Fun and Profit , 2019, 2019 IEEE Symposium on Security and Privacy (SP).

[57]  Wei Song,et al.  Stateful Forward-Edge CFI Enforcement with Intel MPX , 2018, ACA.

[58]  Kai Li,et al.  The PARSEC benchmark suite: Characterization and architectural implications , 2008, 2008 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[59]  Hovav Shacham,et al.  The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86) , 2007, CCS '07.

[60]  Cristiano Giuffrida,et al.  VTPin: practical VTable hijacking protection for binaries , 2016, ACSAC.

[61]  Jonathan M. Smith,et al.  PUMP: a programmable unit for metadata processing , 2014, HASP@ISCA.

[62]  Hui Liu,et al.  New Exploit Methods against Ptmalloc of GLIBC , 2016, 2016 IEEE Trustcom/BigDataSE/ISPA.

[63]  Xi Chen,et al.  A Tough Call: Mitigating Advanced Code-Reuse Attacks at the Binary Level , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[64]  Milo M. K. Martin,et al.  SoftBound: highly compatible and complete spatial memory safety for c , 2009, PLDI '09.

[65]  David A. Wagner,et al.  ROP is Still Dangerous: Breaking Modern Defenses , 2014, USENIX Security Symposium.

[66]  George C. Necula,et al.  CCured: type-safe retrofitting of legacy software , 2005, TOPL.

[67]  Wenke Lee,et al.  Preventing Use-after-free with Dangling Pointers Nullification , 2015, NDSS.

[68]  Gernot Heiser,et al.  CATalyst: Defeating last-level cache side channel attacks in cloud computing , 2016, 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[69]  Derek Bruening,et al.  AddressSanitizer: A Fast Address Sanity Checker , 2012, USENIX Annual Technical Conference.

[70]  Ahmad-Reza Sadeghi,et al.  HardScope: Hardening Embedded Systems Against Data-Oriented Attacks , 2019, 2019 56th ACM/IEEE Design Automation Conference (DAC).

[71]  Calvin Lin,et al.  Efficient and extensible security enforcement using dynamic data flow analysis , 2008, CCS.

[72]  Yunsup Lee,et al.  The RISC-V Instruction Set Manual , 2014 .

[73]  Yuval Yarom,et al.  FLUSH+RELOAD: A High Resolution, Low Noise, L3 Cache Side-Channel Attack , 2014, USENIX Security Symposium.

[74]  Dawn Xiaodong Song,et al.  SoK: Eternal War in Memory , 2013, 2013 IEEE Symposium on Security and Privacy.

[75]  Michael Hamburg,et al.  Meltdown: Reading Kernel Memory from User Space , 2018, USENIX Security Symposium.

[76]  Daniel C. DuVarney,et al.  Efficient Techniques for Comprehensive Protection from Memory Error Exploits , 2005, USENIX Security Symposium.

[77]  Per Larsen,et al.  Losing Control: On the Effectiveness of Control-Flow Integrity under Stack Attacks , 2015, CCS.

[78]  Frank Piessens,et al.  A Systematic Evaluation of Transient Execution Attacks and Defenses , 2018, USENIX Security Symposium.

[79]  Kevin W. Hamlen,et al.  CONFIRM: Evaluating Compatibility and Relevance of Control-flow Integrity Protections for Modern Software , 2019, USENIX Security Symposium.

[80]  Jianping Zhu,et al.  CPU Security Benchmark , 2018 .

[81]  Yuanyuan Zhou,et al.  SafeMem: exploiting ECC-memory for detecting memory leaks and memory corruption during production runs , 2005, 11th International Symposium on High-Performance Computer Architecture.

[82]  Moinuddin K. Qureshi.  New Attacks and Defense for Encrypted-Address Cache , 2019, 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA).

[83]  David A. Wagner,et al.  The Performance Cost of Shadow Stacks and Stack Canaries , 2015, AsiaCCS.

[84]  Berk Sunar,et al.  LVI: Hijacking Transient Execution through Microarchitectural Load Value Injection , 2020, 2020 IEEE Symposium on Security and Privacy (SP).

[85]  Jun Xu,et al.  Non-Control-Data Attacks Are Realistic Threats , 2005, USENIX Security Symposium.

[86]  Milo M. K. Martin,et al.  Hardbound: architectural support for spatial safety of the C programming language , 2008, ASPLOS.

[87]  Wei Song,et al.  Randomized Last-Level Caches Are Still Vulnerable to Cache Side-Channel Attacks! But We Can Fix It , 2020, 2021 IEEE Symposium on Security and Privacy (SP).

[88]  Calton Pu,et al.  Buffer overflows: attacks and defenses for the vulnerability of the decade , 2000, Foundations of Intrusion Tolerant Systems, 2003 [Organically Assured and Survivable Information Systems].

[89]  Joel Emer,et al.  CaSA: End-to-end Quantitative Security Analysis of Randomly Mapped Caches , 2020, 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).