Verifying Instruction Set Simulators using Coverage-guided Fuzzing*

Verification of Instruction Set Simulators (ISSs) is crucial. Predominantly simulation-based approaches are used. They require a comprehensive testset to ensure a thorough verification.We propose a novel coverage-guided fuzzing (CGF) approach to improve the testcase generation process. In addition to code coverage we integrate functional coverage and a custom mutation procedure tailored for ISS verification. As a case-study we apply our approach on a set of three publicly available RISC-V ISSs. We found several new errors, including one error in the official RISC-V reference simulator Spike.

[1]  Sigal Asaf,et al.  FPgen - a test generation framework for datapath floating-point verification , 2003, Eighth IEEE International High-Level Design Validation and Test Workshop.

[2]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[3]  Weiqin Ma,et al.  Rapid prototyping and compact testing of CPU emulators , 2010, Proceedings of 2010 21st IEEE International Symposium on Rapid System Protyping.

[4]  Rolf Drechsler,et al.  Towards early validation of firmware-based power management using virtual prototypes: A constrained random approach , 2017, 2017 Forum on Specification and Design Languages (FDL).

[5]  Avi Ziv,et al.  Coverage directed test generation for functional verification using Bayesian networks , 2003, Proceedings 2003. Design Automation Conference (IEEE Cat. No.03CH37451).

[6]  Rolf Drechsler,et al.  CRAVE: An advanced constrained random verification environment for SystemC , 2012, 2012 International Symposium on System on Chip (SoC).

[7]  Avi Ziv,et al.  Generating instruction streams using abstract CSP , 2012, 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[8]  Rolf Drechsler,et al.  Compiled Symbolic Simulation for SystemC , 2016, 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[9]  Rolf Drechsler,et al.  Extensible and Configurable RISC-V Based Virtual Prototype , 2018, 2018 Forum on Specification & Design Languages (FDL).

[10]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[11]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[12]  Björn Franke,et al.  Automated ISA branch coverage analysis and test case generation for retargetable instruction set simulators , 2014, 2014 International Conference on Compilers, Architecture and Synthesis for Embedded Systems (CASES).

[13]  Brian Campbell,et al.  Randomised testing of a microprocessor model using SMT-solver state generation , 2014, Sci. Comput. Program..

[14]  Rishabh Singh,et al.  Learn&Fuzz: Machine learning for input fuzzing , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[15]  Kerstin Eder,et al.  Feedback-Based Coverage Directed Test Generation: An Industrial Evaluation , 2010, Haifa Verification Conference.

[16]  Allon Adir,et al.  Genesys-Pro: innovations in test program generation for functional processor verification , 2004, IEEE Design & Test of Computers.

[17]  Rolf Drechsler,et al.  Verifying SystemC Using Intermediate Verification Language and Stateful Symbolic Simulation , 2019, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[18]  Lorenzo Martignoni,et al.  Testing CPU emulators , 2009, ISSTA.

[19]  Yehuda Naveh,et al.  X-Gen: a random test-case generator for systems and SoCs , 2002, Seventh IEEE International High-Level Design Validation and Test Workshop, 2002..