An instrumented observability coverage method for system validation

In order to improve effectiveness and efficiency of post-silicon validation, we present a fault-symbol tracking method and a coverage metric that account for the limited observability in silicon and thus are useful for guiding validation test selection, test development, and design for debug. The coverage points targeted in this study are a set of fault-symbols, or ‘tags’, generated from each expression in a system model. Coverage is measured in simulation by tracking tags alongside dynamic information flows to user-defined or implicit observation points. Computation of the metric is performed based on high-level (C/C++) functional and behavioral models through compiler-inserted parallel fault-symbol tracking instrumentation, which offers high efficiency as well as compatibility with existing simulation flows. The coverage results from our initial implementation for a microcontroller instruction set simulator are compared with the statement and mutation coverages. The results show that the new coverage metric is more accurate than the statement coverage and can be computed in significantly shorter runtimes than the mutation coverage.

[1]  Tao Lv,et al.  An Observability Branch Coverage Metric Based on Dynamic Factored Use-Define Chains , 2006, 2006 15th Asian Test Symposium.

[2]  Xiangyu Zhang,et al.  Efficient online detection of dynamic control dependence , 2007, ISSTA '07.

[3]  Ian G. Harris,et al.  A domain coverage metric for the validation of behavioral VHDL descriptions , 2000, Proceedings International Test Conference 2000 (IEEE Cat. No.00CH37159).

[4]  Jonathan P. Bowen,et al.  A formal analysis of MCDC and RCDC test criteria , 2005, Softw. Test. Verification Reliab..

[5]  José Carlos Maldonado,et al.  Proteum/IM 2.0: An Integrated Mutation Testing Environment , 2001 .

[6]  M. Hampton,et al.  Leveraging a Commercial Mutation Analysis Tool For Research , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[7]  Priyadarsan Patra.  On the cusp of a validation wall , 2007, IEEE Design & Test of Computers.

[8]  Tzi-cker Chiueh,et al.  A General Dynamic Information Flow Tracking Framework for Security Applications , 2006, 2006 22nd Annual Computer Security Applications Conference (ACSAC'06).

[9]  Masahiro Fujita,et al.  Event-driven observability enhanced coverage analysis of C programs for functional validation , 2003, ASP-DAC '03.

[10]  Ian G. Harris,et al.  A data flow fault coverage metric for validation of behavioral HDL descriptions , 2000, IEEE/ACM International Conference on Computer Aided Design. ICCAD - 2000. IEEE/ACM Digest of Technical Papers (Cat. No.00CH37140).

[11]  Richard J. Lipton,et al.  Theoretical and empirical studies on using program mutation to test the functional correctness of programs , 1980, POPL '80.

[12]  Chantal Robach,et al.  From specification validation to hardware testing: a unified method , 1996, Proceedings International Test Conference 1996. Test and Design Validity.

[13]  Chien-Nan Jimmy Liu,et al.  An observability measure to enhance statement coverage metric for proper evaluation of verification completeness , 2005, ASP-DAC.

[14]  Elaine J. Weyuker,et al.  Data flow-based test adequacy analysis for languages with pointers , 1991, TAV4.