A probabilistic analysis of coverage methods

Coverage is an important measure for the quality and completeness of the functional verification of hardware logic designs. Verification teams spend a significant amount of time looking for bugs in the design and in providing high-quality coverage. This process is performed through the use of various sampling strategies for selecting test inputs. The selection of sampling strategies to achieve the verification goals is typically carried out in an intuitive manner. We studied several commonly used sampling strategies and provide a probabilistic framework for assessing and comparing their relative values. For this analysis, we derived results for two measures of interest: first, the probability of finding a bug within a given number of samplings; and second, the expected number of samplings until a bug is detected. These results are given for both recurring sampling schemes, in which the same inputs might be selected repeatedly, and for nonrecurring sampling schemes, in which already sampled inputs are never selected again. By considering results from the theory of search, and more specifically, from the well-known multiarmed bandit problem, we demonstrate the optimality of a greedy sampling strategy within our defined framework.

[1]  Todd M. Austin,et al.  Microprocessor Verification via Feedback-Adjusted Markov Models , 2007, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[2]  Sunil Jain,et al.  Statistical Fault Analysis , 1985, IEEE Design & Test of Computers.

[3]  Andrew Piziali,et al.  Functional verification coverage measurement and analysis , 2004 .

[4]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[5]  Vibhav Gogate,et al.  A New Algorithm for Sampling CSP Solutions Uniformly at Random , 2006, CP.

[6]  Ansi Ieee,et al.  IEEE Standard for Binary Floating Point Arithmetic , 1985 .

[7]  Sigal Asaf,et al.  FPgen - a test generation framework for datapath floating-point verification , 2003, Eighth IEEE International High-Level Design Validation and Test Workshop.

[8]  Giovanni Squillero,et al.  Automatic test program generation for pipelined processors , 2003, SAC '03.

[9]  John P. Hayes,et al.  High-level test generation for design verification of pipelined microprocessors , 1999, DAC '99.

[10]  Wolfgang Roesner,et al.  Comprehensive Functional Verification: The Complete Industry Cycle (Systems on Silicon) , 2005 .

[11]  Shmuel Ur,et al.  Micro architecture coverage directed generation of test programs , 1999, DAC '99.

[12]  Vishwani D. Agrawal,et al.  Fault sampling revisited , 1990, IEEE Design & Test of Computers.

[13]  P. Goel.  Test Generation Costs Analysis and Projections , 1980, 17th Design Automation Conference.

[14]  Hans-Joachim Wunderlich.  PROTEST: A Tool for Probabilistic Testability Analysis , 1985, DAC 1985.

[15]  R. Weber.  On the Gittins Index for Multiarmed Bandits , 1992 .

[16]  Stephan Merz,et al.  Model Checking , 2000 .

[17]  Hai Zhou,et al.  Parallel CAD: Algorithm Design and Programming Special Section Call for Papers TODAES: ACM Transactions on Design Automation of Electronic Systems , 2010 .

[18]  Wolfgang Roesner,et al.  Comprehensive Functional Verification: The Complete Industry Cycle , 2005 .

[19]  Guido D. Salvucci,et al.  Ieee standard for binary floating-point arithmetic , 1985 .

[20]  Avi Ziv,et al.  Coverage directed test generation for functional verification using Bayesian networks , 2003, Proceedings 2003. Design Automation Conference (IEEE Cat. No.03CH37451).

[21]  Vishwani D. Agrawal,et al.  STAFAN: An Alternative to Fault Simulation , 1984, 21st Design Automation Conference Proceedings.

[22]  Hans-Joachim Wunderlich,et al.  PROTEST: A Tool for Probabilistic Testability Analysis , 1985, 22nd ACM/IEEE Design Automation Conference.

[23]  Hailong Cui,et al.  Modeling Fault Coverage of Random Test Patterns , 2003, J. Electron. Test..

[24]  Fumiyasu Hirose,et al.  Automatic Test Program Generation For Pipelined Processors , 1994, IEEE/ACM International Conference on Computer-Aided Design.

[25]  Rina Dechter,et al.  Constraint Processing , 1995, Lecture Notes in Computer Science.

[26]  Avi Ziv,et al.  Hole analysis for functional coverage data , 2002, DAC '02.