A structured approach for conducting a series of controlled experiments in software visualization

In the field of software visualization controlled experiments are an important instrument to investigate the specific reasons, why some software visualizations excel the expectations on providing insights and ease task solving while others fail doing so. Despite this, controlled experiments in software visualization are rare. A reason for this is the fact that performing such evaluations in general, and particularly performing them in a way that minimizes the threats to validity, is hard to accomplish. In this paper, we present a structured approach on how to conduct a series of controlled experiments in order to give empirical evidence for advantages and disadvantages of software visualizations in general and of 2D vs. 3D software visualizations in particular.

[1]  Barbara Tversky,et al.  Gratuitous graphics? Putting preferences in perspective , 1996, CHI.

[2]  Wilhelm Hasselbring,et al.  Synchrovis: 3D visualization of monitoring traces in the city metaphor for analyzing concurrency , 2013, 2013 First IEEE Working Conference on Software Visualization (VISSOFT).

[3]  Stephan Diehl,et al.  Software Visualization - Visualizing the Structure, Behaviour, and Evolution of Software , 2007 .

[4]  Victor R. Basili,et al.  Experimentation in software engineering , 1986, IEEE Transactions on Software Engineering.

[5]  Tobias Isenberg,et al.  A Systematic Review on the Practice of Evaluating Visualization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[6]  Orla Greevy.  Dynamix - a Meta-Model to Support Feature-Centric Analysis , 2007, FAMOOSr@TOOLS.

[7]  Brad A. Myers,et al.  Taxonomies of visual programming and program visualization , 1990, J. Vis. Lang. Comput..

[8]  Colin Ware,et al.  Visualizing object oriented software in three dimensions , 1993, CASCON.

[9]  Romain Robbes,et al.  Software systems as cities: a controlled experiment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[10]  Blaine A. Price,et al.  A Principled Taxonomy of Software Visualization , 1993, J. Vis. Lang. Comput..

[11]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[12]  Colin Ware,et al.  Viewing a graph in a virtual reality display is three times as good as a 2D diagram , 1994, Proceedings of 1994 IEEE Symposium on Visual Languages.

[13]  Catherine Plaisant,et al.  Visualizing Graphs as Trees: Plant a Seed and Watch it Grow , 2005, Graph Drawing.

[14]  Pourang Irani,et al.  Diagramming information structures using 3D perceptual primitives , 2003, TCHI.

[15]  Marcus Barkowsky,et al.  The Importance of Visual Attention in Improving the 3D-TV Viewing Experience: Overview and New Perspectives , 2011, IEEE Transactions on Broadcasting.

[16]  Stéphane Ducasse,et al.  Modeling Software Evolution by Treating History as a First Class Entity , 2005, Electron. Notes Theor. Comput. Sci..

[17]  Eileen Kraemer,et al.  Designing your Next Empirical Study on Program Comprehension , 2007, 15th IEEE International Conference on Program Comprehension (ICPC '07).

[18]  Marcelo R. Campo,et al.  An Overview of 3D Software Visualization , 2009, IEEE Transactions on Visualization and Computer Graphics.

[19]  Malcolm Munro,et al.  Software Architecture Visualization: An Evaluation Framework and Its Application , 2008, IEEE Transactions on Software Engineering.

[20]  D. Leutner,et al.  Instructional animation versus static pictures: A meta-analysis , 2007 .

[21]  Pierre Poulin,et al.  Visualizing software dynamicities with heat maps , 2013, 2013 First IEEE Working Conference on Software Visualization (VISSOFT).

[22]  Hideki Koike,et al.  How does 3-D visualization work in software engineering?: empirical study of a 3-D version/module visualization system , 1998, Proceedings of the 20th International Conference on Software Engineering.

[23]  Igor G. Olaizola,et al.  HTML5-based System for Interoperable 3D Digital Home Applications , 2012 .

[24]  Marin Litoiu,et al.  An Architecture to Support Model Driven Software Visualization , 2006, 14th IEEE International Conference on Program Comprehension (ICPC'06).

[25]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[26]  Colin Ware,et al.  Evaluating stereo and motion cues for visualizing information nets in three dimensions , 1996, TOGS.

[27]  Andy Cockburn,et al.  3D or not 3D?: evaluating the effect of the third dimension in a document management system , 2001, CHI.

[28]  Árpád Beszédes,et al.  CodeMetrpolis — A minecraft based collaboration tool for developers , 2013, 2013 First IEEE Working Conference on Software Visualization (VISSOFT).

[29]  Walter F. Tichy,et al.  Empirische Methodik in der Softwaretechnik im Allgemeinen und bei der Software-Visualisierung im Besonderen , 2007, Software Engineering.

[30]  Wilhelm Hasselbring,et al.  Live trace visualization for comprehending large software landscapes: The ExplorViz approach , 2013, 2013 First IEEE Working Conference on Software Visualization (VISSOFT).

[31]  John Stasko,et al.  Understanding and Characterizing Program Visualization Systems , 1991 .

[32]  Daniel A. Keim,et al.  Mastering the Information Age - Solving Problems with Visual Analytics , 2010 .

[33]  Patrick Ogao,et al.  Evaluation of software visualization tools: Lessons learned , 2009, 2009 5th IEEE International Workshop on Visualizing Software for Understanding and Analysis.

[34]  Shari Lawrence Pfleeger,et al.  Experimental design and analysis in software engineering , 1995, Ann. Softw. Eng..

[35]  Andrian Marcus,et al.  A task oriented view of software visualization , 2002, Proceedings First International Workshop on Visualizing Software for Understanding and Analysis.

[36]  Tamara Munzner,et al.  The four-level nested model revisited: blocks and guidelines , 2012, BELIV '12.

[37]  Andy Cockburn,et al.  Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments , 2002, CHI.

[38]  Victoria Interrante,et al.  User Studies: Why, How, and When? , 2003, IEEE Computer Graphics and Applications.

[39]  Tomás Pajdla,et al.  3D with Kinect , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[40]  Gruia-Catalin Roman,et al.  A taxonomy of program visualization systems , 1993, Computer.

[41]  Pourang Irani,et al.  Diagrams based on structural object perception , 2000, AVI '00.

[42]  Janet Siegmund,et al.  Framework for measuring program comprehension , 2012 .

[43]  Ivan Herman,et al.  Graph Visualization and Navigation in Information Visualization: A Survey , 2000, IEEE Trans. Vis. Comput. Graph..

[44]  Daniel M. German,et al.  On the use of visualization to support awareness of human activities in software development: a survey and a framework , 2005, SoftVis '05.

[45]  Tim Dwyer,et al.  Three Dimensional UML Using Force Directed Layout , 2001, InVis.au.

[46]  John T. Stasko,et al.  Low-level components of analytic activity in information visualization , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[47]  Mohamed Eltoweissy,et al.  Software visualization , 2005, Innovations in Systems and Software Engineering.

[48]  John T. Stasko,et al.  Toward a Deeper Understanding of the Role of Interaction in Information Visualization , 2007, IEEE Transactions on Visualization and Computer Graphics.

[49]  Keith Andrews.  Evaluation Comes in Many Guises , 2008 .

[50]  Oscar Nierstrasz,et al.  The story of moose: an agile reengineering environment , 2005, ESEC/FSE-13.

[51]  Colin Ware,et al.  Visualizing graphs in three dimensions , 2008, TAP.

[52]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[53]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[54]  Bonita Sharif,et al.  An empirical study assessing the effect of seeit 3D on comprehension , 2013, 2013 First IEEE Working Conference on Software Visualization (VISSOFT).

[55]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[56]  J.T. Stasko,et al.  Three-dimensional computation visualization , 1993, Proceedings 1993 IEEE Symposium on Visual Languages.

[57]  Pascal Kovacs,et al.  Generative Software Visualization : Automatic Generation of User-Specific Visualizations , .