An experimental study of the impact of visual semantic feedback on novice programming

Prior empirical studies of programming have shown that novice programmers tend to program by exploration, relying on frequent compilation and execution of their code in order to make progress. One way visual and end-user programming environments have attempted to facilitate this exploratory programming process is through their support of ''live'' editing models, in which immediate visual feedback on a program's execution is provided automatically at edit time. Notice that the notion of ''liveness'' actually encompasses two distinct dimensions: (a) the amount of time a programmer must wait between editing a program and receiving visual feedback (feedback delay); and (b) whether such feedback is provided automatically, or whether the programmer must explicitly request it (feedback self-selection). While a few prior empirical studies of ''live'' editing do exist, none has specifically evaluated the impact of these dimensions of ''live'' editing within the context of the imperative programming paradigm commonly taught in first-semester computer science courses. As a preliminary step toward that end, we conducted an experimental study that investigated the impact of feedback self-selection on novice imperative programming. Our within-subjects design compared the impact of three different levels of feedback self-selection on syntactic and semantic correctness: (a) no visual feedback at all (the No Feedback treatment); (b) visual feedback, in the form of a visualization of the program's execution state, provided on request when a ''run'' button is hit (the Self-Select treatment); and (c) visual feedback, in the form of a visualization of the program's execution state, updated on every keystroke (the Automatic treatment). Participants in the Automatic and Self-Select treatments produced programs that had significantly fewer syntactic and semantic errors than those of the No Feedback treatment; however, no significant differences were found between the Automatic and Self-Select treatments. These results suggest that, at least in the case of novice imperative programming environments, the benefits of delivering a continuously updated visual representation of a program's execution may fail to justify the substantial costs of implementing such feedback. We recommend that programming environment designers instead direct their efforts toward carefully considering when programmers will be ready to take advantage of the feedback that is coming toward them, along with what content will be of most benefit to them.

[1]  Jonathan J. Cadiz,et al.  Does continuous visual feedback aid debugging in direct-manipulation programming systems? , 1997, CHI.

[2]  Henry C. Lucas,et al.  System response time operator productivity, and job satisfaction , 1983, CACM.

[3]  Steven L. Tanimoto,et al.  VIVA: A visual language for image processing , 1990, J. Vis. Lang. Comput..

[4]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[5]  Steven M. Hadfield,et al.  RAPTOR: a visual programming environment for teaching algorithmic problem solving , 2005, SIGCSE '05.

[6]  Randy F. Pausch,et al.  Making the connection: programming with animated small world , 2000, ITiCSE '00.

[7]  Brad A. Myers,et al.  Design requirements for more flexible structured editors from a study of programmers' text editing , 2005, CHI Extended Abstracts.

[8]  Wesley J. Chun.  Core Python Programming , 2000 .

[9]  Shaaron Ainsworth,et al.  Examining the Effects of Different Multiple Representational Systems in Learning Primary Mathematics , 2002 .

[10]  Christopher D. Hundhausen,et al.  What you see is what you code: a radically dynamic algorithm visualization development model for novice learners , 2005, 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'05).

[11]  Kate Ehrlich,et al.  Empirical Studies of Programming Knowledge , 1984, IEEE Transactions on Software Engineering.

[12]  John R. Anderson,et al.  Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes , 2001, CHI.

[13]  B. Schneirdeman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[14]  Hans Spada,et al.  The Active Integration of Information during Learning with Dynamic and Interactive Visualisations , 2004 .

[15]  Ben Shneiderman,et al.  Designing the user interface (2nd ed.): strategies for effective human-computer interaction , 1992 .

[16]  J. Sweller,et al.  Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions , 2005 .

[17]  Mitchell Grossberg,et al.  An Experiment on Problem Solving with Delayed Computer Responses , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[18]  Michael Kölling,et al.  The BlueJ System and its Pedagogy , 2003, Comput. Sci. Educ..

[19]  Ben Shneiderman,et al.  Direct Manipulation: A Step Beyond Programming Languages , 1983, Computer.

[20]  Mordechai Ben-Ari,et al.  The Jeliot 2000 program animation system , 2003, Comput. Educ..

[21]  Ben Shneiderman,et al.  Directions in human-computer interaction , 1982 .

[22]  R. Kent Dybvig,et al.  Revised5 Report on the Algorithmic Language Scheme , 1986, SIGP.

[23]  Margaret Burnett,et al.  A bug's eye view of immediate visual feedback in direct-manipulation programming systems , 1997, ESP '97.

[24]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[25]  Gale L. Martin,et al.  System response time effects on user productivity , 1986 .

[26]  Thomas Reps,et al.  Programming Techniques and Data Structures , 1981 .

[27]  Tina Seufert.  Supporting Coherence Formation in Learning from Multiple Representations , 2003 .

[28]  Margaret M. Burnett,et al.  Interactive Visual Data Abstraction in a Declarative Visual Programming Language , 1994, J. Vis. Lang. Comput..

[29]  Peter Brusilovsky,et al.  Mini-languages: a way to learn programming principles , 2004, Education and Information Technologies.

[30]  Philip Miller,et al.  Evolution of Novice Programming Environments: The Structure Editors of Carnegie Mellon University , 1994, Interact. Learn. Environ..