An Investigation into Freeform, Dynamic, Digital Ink Annotation for Program Code

Understanding program code is cognitively demanding. One tool that has not been investigated previously is using ink freeform annotations to aid understanding of program code. Before we can investigate using freeform annotations for code comprehension there are some user and technical challenges that need answering. An iterative research approach was used for this investigation. A single research question was investigated in each iteration and the results informed the next iteration. In the first iteration, the focus was how and why programmers annotate when understanding program code on paper. The results indicate programmers reading program code use similar types of annotation to those reported for other types of reading. The main reasons for adding annotations were to assist with navigation and offload information from the reader’s memory. In the second iteration, the focus was how to classify digital annotations; a precursor to other operations. The initial phase used a general purpose automatic recogniser but failed to improve on previously reported results. The second phase combined automatic recognition with user input. While this approach resulted in higher classification accuracy, the accuracy rates were still lower than previously reported for sketch-based recognition. An unexpected finding was the participants did not want to classify all annotations during reading. In the third iteration, the focus was how to refit (modify) annotations in response to changes in the underlying text. Four classes of annotation were investigated: horizontal lines, vertical lines, enclosures and connectors. Several refitting algorithms were implemented and evaluated for each class of annotation. The findings indicate there are two preferred approaches for refitting annotations: stretching the annotation or splitting it and adding a visualisation showing where the annotation was split. This thesis makes five main contributions. First, a systematic literature review which provides an overview of the current research. Second, details of how and why programmers annotate code on paper. Third, an implementation of an extensible tool for investigating digital ink annotations on code. Fourth, details on how collaborative intelligence can improve recognition. Fifth, a set of proposals for how to refit annotations based on the annotation classification and user preferences.