Toward a theory of critiquing human errors in computer programming

The purpose of this research is to pose and test various theories of expert critics in computer programming. The goal is to find out how a computer critic can best impact programmers' behavior during programming tasks. Three empirical studies were conducted. The first study consisted of the control group. Its purpose was to find syntax errors that programmers make when programming. To this end, subjects were given a programming task. Thirteen different kinds of syntax errors were made by the subjects; errors per subject ranged from one to eleven. These results revealed not only different kinds of syntax errors, but also the syntax errors common to most subjects. In the second study, subjects were given the same programming task, but this time had the assistance of a critic. The purpose of this critic was to prevent, detect and correct syntax errors during programming tasks. Several dozen critic networks were implemented based on the following critiquing theory: (1) timing of the critic was before, during and after task, (2) process of the critic was incremental, (3) mode of the critic was active, (4) knowledge in the critic was both deep and shallow, (5) heuristics algorithms were used, (6) interface of the critic was direct manipulation driven, and (7) strategies used in the critic were influencers, directors, and debiasers. The results of the second study showed that critics reduced the programmers' syntax errors by 85%. The only drawback was that the critic slowed programmers down. Therefore, experienced programmers disliked using the critic, although novice programmers found it helpful during programming tasks. The third study was conducted to evaluate both the effectiveness of the critic's textual information and its usability. To evaluate effectiveness of the textual information, the critic only signaled users if an error occurred, but did not provide the source of the error. These results showed that subjects performed much better with textual information. In addition, a survey questionnaire designed to evaluate the critic's usability and effectiveness and the best critiquing approach and strategies in computer programming was completed by subjects. This questionnaire also evaluated the software covering keystrokes, clarity, and sequence of operations of the critic. These results showed that users like to be critiqued; (1) during and after task, (2) incrementally, and (3) actively. Regarding critiquing strategies, the subjects liked directors and debiasers, but only liked influencers which dealt with cautions regarding the next action.