Does syntax highlighting help programming novices?

Program comprehension is an important skill for programmers – extending and debugging existing source code is part of the daily routine. Syntax highlighting is one of the most common tools used to support developers in understanding algorithms. However, most research in this area originates from a time when programmers used a completely different tool chain. We examined the influence of syntax highlighting on novices’ ability to comprehend source code. Additional analyses cover the influence of task type and programming experience on the code comprehension ability itself and its relation to syntax highlighting. We conducted a controlled experiment with 390 undergraduate students in an introductory Java programming course. We measured the correctness with which they solved small coding tasks. Each test subject received some tasks with syntax highlighting and some without. The data provided no evidence that syntax highlighting improves novices’ ability to comprehend source code. There are very few similar experiments and it is unclear as of yet which factors impact the effectiveness of syntax highlighting. One major limitation may be the types of tasks chosen for this experiment. The results suggest that syntax highlighting squanders a feedback channel from the IDE to the programmer that can be used more effectively.

[1]  Matthew Flatt,et al.  ProfessorJ: a gradual introduction to Java through language levels , 2003, OOPSLA '03.

[2]  W. Allen Wallis,et al.  Compounding Probabilities from Independent Significance Tests , 1942 .

[3]  Lutz Prechelt,et al.  The 28:1 Grant/Sackman legend is misleading, or: How large is interpersonal variation really? , 1999 .

[4]  Curtis R. Cook,et al.  Typographic style is more than cosmetic , 1990, CACM.

[5]  Jorma Sajaniemi,et al.  An Experiment on the Effects of Program Code Highlighting on Visual Search for Local Patterns , 2006, PPIG.

[6]  Susan Bergin,et al.  Programming: factors that influence success , 2005 .

[7]  Rebecca Tiarks.  What Programmers Really Do - An Observational Study , 2011, Softwaretechnik-Trends.

[8]  Jan Mendling,et al.  Syntax highlighting in business process models , 2011, Decis. Support Syst..

[9]  Robert Cartwright,et al.  DrJava: a lightweight pedagogic environment for Java , 2002, SIGCSE '02.

[10]  Curtis R. Cook,et al.  Does Signaling Help Professional Programmers Read and Understand Computer Programs , 1991 .

[11]  Sven Apel,et al.  Measuring and modeling programming experience , 2013, Empirical Software Engineering.

[12]  John Ludbrook,et al.  Analysis of 2 x 2 tables of frequencies: matching test to experimental design. , 2008, International journal of epidemiology.

[13]  Dietmar Pfahl,et al.  Reporting Experiments in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[14]  J. Moran,et al.  Sensation and perception , 1980 .

[15]  Natalia Juristo Juzgado,et al.  Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study , 2017, Empirical Software Engineering.

[16]  E. E. Grant,et al.  Exploratory experimental studies comparing online and offline programming performance , 1968, CACM.

[17]  Thomas Leich,et al.  Do background colors improve program comprehension in the #ifdef hell? , 2012, Empirical Software Engineering.

[18]  E. Burton Swanson,et al.  Characteristics of application software maintenance , 1978, CACM.

[19]  Tanya Beelders,et al.  Syntax highlighting as an influencing factor when reading and comprehending source code , 2015 .

[20]  Pierre Dragicevic,et al.  Fair Statistical Communication in HCI , 2016 .

[21]  Robert W. Bowdidge,et al.  Programmers' build errors: a case study (at google) , 2014, ICSE.

[22]  Rick Kazman,et al.  Determining the usefulness of colour and fonts in a programming task , 1994, Proceedings 1994 IEEE 3rd Workshop on Program Comprehension- WPC '94.

[23]  Donald E. Knuth,et al.  Literate Programming , 1984, Comput. J..

[24]  H. Sackman,et al.  An Exploratory Investigation of Programmer Performance Under On-Line and Off-Line Conditions , 1967 .

[25]  Martha E. Crosby,et al.  How do we read algorithms? A case study , 1990, Computer.

[26]  R. Mehta,et al.  Blue or Red? Exploring the Effect of Color on Cognitive Task Performances , 2009, Science.

[27]  Giovanna Maria Dimitri.  The impact of Syntax Highlighting in Sonic Pi , 2015, PPIG.

[28]  Gerard K. Rambally.  The influence of color on program readability and comprehensibility , 1986 .

[29]  Aaron Marcus,et al.  Human factors and typography for more readable programs , 1990 .

[30]  Edward Holden,et al.  The impact of prior experience in an information technology programming course sequence , 2003, CITC4 '03.

[31]  Darren Van Laar.  Evaluating a colour coding programming support tool , 1990 .

[32]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[33]  Wilkie W. Chaffin,et al.  The effect of skewness and kurtosis on the one-sample T test and the impact of knowledge of the population standard deviation , 1993 .

[34]  Andrew A. Berlin,et al.  Compiling scientific code using partial evaluation , 1990, Computer.

[35]  Volker Gruhn,et al.  Using Wikis as Software Development Environments , 2012, SoMeT.

[36]  Advait Sarkar,et al.  The impact of syntax colouring on program comprehension , 2015, PPIG.

[37]  Harald C. Gall,et al.  Don't touch my code!: examining the effects of ownership on software quality , 2011, ESEC/FSE '11.

[38]  Sebastian Kleinschmager,et al.  How to rate programming skills in programming experiments?: a preliminary, exploratory, study based on university marks, pretests, and self-estimation , 2011, PLATEAU '11.