Evaluating Program Visualisation Systems : An Information-Based Methodology

A number of Program Visualisation (PV) systems have been developed to aid the debugging process by providing a detailed account of program execution. Despite their development, empirical claims for their efficacy are few. A review of related empirical evaluations identifies the lack of a principled and integrated approach as being a significant problem. This criticism combined with a description of SV within the context of the programming environment is used to develop a set of methodological criteria for future work. These requirements have motivated the implementation of the Prolog Program Visualization Laboratory (PPVL) which provides an integrated tracing environment comprising four Prolog tracers. An information-based methodology is then presented for evaluating Prolog tracers within the realistic programming environment provided by PPVL. Finally, a survey of PV design claims are used to focus future research.

[1]  Tim Rajan,et al.  Principles for the design of dynamic tracing environments for novice programmers , 1990 .

[2]  Rachel K. E. Bellamy,et al.  Parsing and Gnisrap: a model of device use , 1987 .

[3]  David J. Gilmore.  Does the Notation Matter , 1994 .

[4]  Marc Eisenstadt,et al.  The Transparent Prolog Machine (TPM): An Execution Model and Graphical Debugger for Logic Programming , 1988, J. Log. Program..

[5]  T. R. G. Green.  Conditional program statements and their comprehensibility to professional programmers , 1977 .

[6]  John T. Stasko,et al.  Do algorithm animations assist learning?: an empirical study and analysis , 1993, INTERCHI.

[7]  Thomas R. G. Green,et al.  Scope Marking in Computer Conditionals - A Psychological Evaluation , 1977, Int. J. Man Mach. Stud..

[8]  Joseph W. Rigney,et al.  Pictorial Presentation and Review Strategies in Science Learning. , 1981 .

[9]  John R. Anderson,et al.  A model of novice debugging in LISP , 1986 .

[10]  Christopher J. Taylor,et al.  Textual Tree (Prolog) Tracer: An Experimental Evaluation , 1994 .

[11]  Kristina Höök,et al.  Redo “Try once and pass”: the influence of complexity and graphical notation on novices' understanding of Prolog , 1990 .

[12]  T. Rajan.  A principled design for an animated view of program execution for novice programmers , 1986 .

[13]  Stephen K. Reed,et al.  Effect of computer graphics on improving estimates to algebra word problems. , 1985 .

[14]  Alan Bundy,et al.  What stories should we tell novice PROLOG programmers , 1987 .

[15]  Marc Eisenstadt,et al.  A Practical Graphical Tracer for Prolog , 1991, Int. J. Man Mach. Stud..

[16]  Marc Eisenstadt,et al.  A Powerful Prolog Trace Package , 1984, European Conference on Artificial Intelligence.

[17]  Stephen J. Payne,et al.  Algebra Mal‐Rules and Cognitive Accounts of Error , 1990 .

[18]  Nancy Pennington,et al.  Comprehension strategies in programming , 1987 .

[19]  Marian Petre,et al.  Learning to Read Graphics: Some Evidence that 'Seeing' an Information Display is an Acquired Skill , 1993, J. Vis. Lang. Comput..

[20]  Maarten van Someren,et al.  What's wrong? Understanding beginners' problems with Prolog , 1990 .

[21]  Iris Vessey,et al.  Expertise in Debugging Computer Programs: An Analysis of the Content of Verbal Protocols , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[22]  John R. Anderson,et al.  Debugging: An Analysis of Bug-Location Strategies , 1987, Hum. Comput. Interact..

[23]  Edward A. Youngs.  Human Errors in Programming , 1974, Int. J. Man Mach. Stud..

[24]  Kate Ehrlich,et al.  Empirical Studies of Programming Knowledge , 1984, IEEE Transactions on Software Engineering.

[25]  Josephine Annette Taylor.  Programming in prolog: an indepth study of problems for beginners learning to program in prolog , 1988 .

[26]  Mike Brayshaw,et al.  Towards a taxonomy of novices' misconceptions of the Prolog interpreter , 1990 .

[27]  Simon P. Davies,et al.  Effects of concurrent verbalization on design problem solving , 1995 .

[28]  Nancy Cunniff,et al.  Graphical vs. textual representation: an empirical study of novices' program comprehension , 1987 .

[29]  Fred P. Brooks,et al.  The Mythical Man-Month , 1975, Reliable Software.

[30]  Lucy A. Suchman,et al.  Plans and Situated Actions: The Problem of Human-Machine Communication (Learning in Doing: Social, , 1987 .

[31]  Albert N. Badre,et al.  Graphic language representation and programming behavior , 1989 .

[32]  H. J. Peters,et al.  Graphics and Animation as Instructional Tools: A Case Study. , 1982 .

[33]  Johnette Hassell,et al.  Information Relationships in PROLOG Programs: How Do Programmers Comprehend Functionality? , 1991, Int. J. Man Mach. Stud..

[34]  Benjamin H. Layne,et al.  Color, graphics, and animation in a computer-assisted learning tutorial lesson , 1988 .

[35]  Benedict du Boulay,et al.  Outline Proposal for a Prolog `Textual Tree Tracer' (TTT) , 1991 .

[36]  Zahava Scherz,et al.  Cognitive Implications of Learning Prolog—Mistakes and Misconceptions , 1990 .

[37]  Tim Rajan.  An evaluation of APT: an Animated Program Tracer for novice Prolog programmers , 1991 .

[38]  Kathy A. Lutz,et al.  Effect of graphic analogies of concepts in chemistry on learning and attitudes. , 1976 .

[39]  John R. Anderson,et al.  Knowledge Transfer among Programming Languages , 1992, PPIG.

[40]  Marcel Adam Just,et al.  The Role of Working Memory in Language Comprehension , 1988 .

[41]  Peter Brusilovsky.  Program visualization as a debugging tool for novices , 1993, CHI '93.

[42]  Gary M. Olson,et al.  Comprehension differences in debugging by skilled and novice programmers , 1986 .

[43]  K. Gilhooly,et al.  Lines of thinking : reflections on the psychology of thought , 1990 .

[44]  Benedict du Boulay,et al.  Effect of Format on Information and Problem Solving , 1991 .

[45]  Elliot Soloway,et al.  Analyzing the high frequency bugs in novice programs , 1986 .

[46]  Elliot Soloway,et al.  Empirical Studies of Programmers: Second Workshop , 1991 .