Towards Assessing the Readability of Programming Error Messages

Programming error messages have proven to be notoriously problematic for novices who are learning to program. Although recent efforts have focused on improving message wording, these have been criticized for attempting to improve usability without first understanding and addressing readability. To date, there has been no research dedicated to the readability of programming error messages and how this could be assessed. In this paper we examine human-based assessments of programming error message readability and make two important contributions. First, we conduct an experiment using the top twenty most-frequent error messages in three popular programming languages (Python, Java, and C), revealing that human notions of readability are highly subjective and dependent on both programming experience and language familiarity. Both novices and experts agreed more about which messages are more readable, but disagreed more about which messages are not readable. Second, we leverage the data from this experiment to uncover several key factors that seem to affect message readability: message length, message tone, and use of jargon. We discuss how these factors can help guide future efforts to design a readability metric for programming error messages.

[1]  Tetsu Iwata,et al.  Symmetric Cryptography (Dagstuhl Seminar 18021) , 2018, Dagstuhl Reports.

[2]  Brett A. Becker,et al.  Categorizing Compiler Error Messages with Principal Component Analysis , 2016 .

[3]  Shriram Krishnamurthi,et al.  Human-Centric Development of Software Tools (Dagstuhl Seminar 15222) , 2015, Dagstuhl Reports.

[4]  Kathi Fisler,et al.  Mind your language: on novices' interactions with error messages , 2011, Onward! 2011.

[5]  Brett A. Becker,et al.  Effective compiler error message enhancement for novice programming students , 2016, Comput. Sci. Educ..

[6]  Ewan D. Tempero,et al.  Understanding the syntax barrier for novices , 2011, ITiCSE '11.

[7]  Frederick W. B. Li,et al.  No tests required: comparing traditional and dynamic predictors of programming success , 2014, SIGCSE.

[8]  Saul Rosen,et al.  PUFFT—The Purdue University fast FORTRAN translator , 1965, CACM.

[9]  Richard L. Wexelblat,et al.  Maxims for malfeasant designers, or how to design languages to make programming as difficult as possible , 1976, ICSE '76.

[10]  V. Javier Traver,et al.  On Compiler Error Messages: What They Say and What They Mean , 2010, Adv. Hum. Comput. Interact..

[11]  Brett A. Becker.  An Exploration Of The Effects Of Enhanced Compiler Error Messages For Computer Programming Novices , 2015 .

[12]  David Pritchard,et al.  Frequency distribution of error messages , 2015, PLATEAU@SPLASH.

[13]  Shriram Krishnamurthi,et al.  Error messages are classifiers: a process to design and evaluate error messages , 2017, Onward!.

[14]  Matthew C. Jadud,et al.  An exploration of novice compilation behaviour in BlueJ , 2006 .

[15]  Andrew Luxton-Reilly,et al.  Enhancing syntax error messages appears ineffectual , 2014, ITiCSE '14.

[16]  Ron Coleman.  Aesthetics Versus Readability of Source Code , 2018 .

[17]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[18]  Raymond P. L. Buse,et al.  A metric for software readability , 2008, ISSTA '08.

[19]  Cristina Giannantonio,et al.  Book Review: Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Thousand Oaks, CA: Sage , 2010 .

[20]  Brett A. Becker,et al.  Research This! Questions that Computing Educators Most Want Computing Education Researchers to Answer , 2019, ICER.

[21]  Stefan Hanenberg,et al.  Evidence About Programmers for Programming Language Design (Dagstuhl Seminar 18061) , 2018, Dagstuhl Reports.

[22]  John Homer,et al.  Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive. , 2017, SIGCSE.

[23]  Brett A. Becker.  An Effective Approach to Enhancing Compiler Error Messages , 2016, SIGCSE.

[24]  Peter C. Rigby,et al.  Study of novice programmers using Eclipse and Gild , 2005, eclipse '05.

[25]  Brett A. Becker,et al.  What Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students? , 2019, SIGCSE.

[26]  Brett A. Becker,et al.  Error Message Readability and Novice Debugging Performance , 2020, ITiCSE.

[27]  Peter J. Brown,et al.  Error messages: the neglected area of the man/machine interface , 1983, CACM.

[28]  Brett A. Becker,et al.  The Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior , 2020, SIGCSE.

[29]  Davin McCall.  Novice programmer errors: analysis and diagnostics , 2016 .

[30]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[31]  Peter J. Brown.  ‘My system gives excellent error messages’—or does it? , 1982, Softw. Pract. Exp..

[32]  Thomas R. Flowers,et al.  Empowering students and building confidence in novice programmers through Gauntlet , 2004, 34th Annual Frontiers in Education, 2004. FIE 2004..

[33]  Titus Barik,et al.  Error Messages as Rational Reconstructions. , 2018 .

[34]  Amey Karkare,et al.  Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research , 2019, ITiCSE-WGR.

[35]  Neil Brown,et al.  Investigating novice programming mistakes: educator beliefs vs. student data , 2014, ICER '14.

[36]  Janet Siegmund,et al.  Shorter identifier names take longer to comprehend , 2017, 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[37]  Ewan D. Tempero,et al.  All syntax errors are not equal , 2012, ITiCSE '12.

[38]  John Homer,et al.  On Novices' Interaction with Compiler Error Messages: A Human Factors Approach , 2017, ICER.

[39]  Elliot Soloway,et al.  Novice mistakes: are the folk wisdoms correct? , 1986, CACM.

[40]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[41]  Emerson R. Murphy-Hill,et al.  How should compilers explain problems to developers? , 2018, ESEC/SIGSOFT FSE.

[42]  Amy J. Ko,et al.  Personifying programming tool feedback improves novice programmers' learning , 2011, ICER.

[43]  Abram Hindle,et al.  Syntax and Stack Overflow: A Methodology for Extracting a Corpus of Syntax Errors and Fixes , 2019, 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[44]  Stephen H. Edwards,et al.  ProgSnap2: A Flexible Format for Programming Process Data , 2019, ITiCSE.

[45]  Emerson R. Murphy-Hill,et al.  Do Developers Read Compiler Error Messages? , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[46]  Ben Shneiderman,et al.  Designing computer system messages , 1982, CACM.

[47]  James J. Horning,et al.  What the Compiler Should Tell the User , 1976, Compiler Construction.