Enabling Visual Design Verification Analytics - From Prototype Visualizations to an Analytics Tool using the Unity Game Engine

The ever-increasing architectural complexity in contemporary ASIC projects turns Design Verification (DV) into a highly advanced endeavor. Pressing needs for short time-to-market has made automation a key solution in DV. However, recurring execution of large regression suites inevitably leads to challenging amounts of test results. Following the design science paradigm, we present an action research study to introduce visual analytics in a commercial ASIC project. We develop a cityscape visualization tool using the game engine Unity. Initial evaluations are promising, suggesting that the tool offers a novel approach to identify error-prone parts of the design, as well as coverage holes.

[1]  Romain Robbes,et al.  Software systems as cities: a controlled experiment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[2]  Daniel A. Keim,et al.  Visual Analytics , 2009, Encyclopedia of Database Systems.

[3]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[4]  Per Runeson,et al.  Supporting Regression Test Scoping with Visual Analytics , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[5]  Markus Borg,et al.  An Analytical View ofTest Results Using CityScapes , 2018 .

[6]  Artur Sosnówka Test City metaphor as support for visual testcase analysis within integration test domain , 2013, 2013 Federated Conference on Computer Science and Information Systems.

[7]  Tibor Gyimóthy,et al.  Using the City Metaphor for Visualizing Test-Related Metrics , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[8]  N. Nagappan,et al.  Use of relative code churn measures to predict system defect density , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[9]  J. Wyatt,et al.  Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide , 2014, BMJ : British Medical Journal.

[10]  Per Runeson,et al.  Using a Visual Abstract as a Lens for Communicating and Promoting Design Science Research in Software Engineering , 2017, 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[11]  Christoph Treude,et al.  Awareness 2.0: staying aware of projects, developers and tasks using dashboards and feeds , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[12]  J. Aken Management Research as a Design Science: Articulating the Research Products of Mode 2 Knowledge Production in Management , 2005 .

[13]  Daniel Hansson Automatic Bug Fixing , 2015, 2015 16th International Workshop on Microprocessor and SOC Test and Verification (MTV).

[14]  Árpád Beszédes,et al.  CodeMetropolis - code visualisation in MineCraft , 2013, 2013 IEEE 13th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[15]  Mohammad Ghafari,et al.  CityVR: Gameful Software Visualization , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[16]  Roel Wieringa,et al.  Technical Action Research , 2014 .

[17]  Michele Lanza,et al.  Visualizing Software Systems as Cities , 2007, 2007 4th IEEE International Workshop on Visualizing Software for Understanding and Analysis.

[18]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[19]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[20]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[21]  Harry D. Foster Trends in functional verification: A 2014 industry study , 2015, 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[22]  Claes Wohlin,et al.  Towards a decision-making structure for selecting a research design in empirical software engineering , 2014, Empirical Software Engineering.

[23]  Arie van Deursen,et al.  Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining , 2008, Empirical Software Engineering.

[24]  Michele Lanza,et al.  Manhattan — 3D City Visualizations in Eclipse , 2011 .

[25]  Ian Alexander,et al.  Handbook of Action Research Participative Inquiry and Practice , 2001, Eur. J. Inf. Syst..

[26]  Per Runeson,et al.  Navigating Information Overload Caused by Automated Testing - a Clustering Approach in Multi-Branch Development , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[27]  H. Bradbury,et al.  Handbook of action research : participative inquiry and practice , 2001 .

[28]  Stephan Diehl,et al.  Software Visualization - Visualizing the Structure, Behaviour, and Evolution of Software , 2007 .

[29]  Edward R. Tufte,et al.  The Visual Display of Quantitative Information , 1986 .