Error messages are classifiers: a process to design and evaluate error messages

This paper presents a lightweight process to guide error report authoring. We take the perspective that error reports are really classifiers of program information. They should therefore be subjected to the same measures as other classifiers (e.g., precision and recall). We formalize this perspective as a process for assessing error reports, describe our application of this process to an actual programming language, and present a preliminary study on the utility of the resulting error reports.

[1]  Amy J. Ko,et al.  Personifying programming tool feedback improves novice programmers' learning , 2011, ICER.

[2]  Kathi Fisler,et al.  WeScheme: the browser is your programming environment , 2011, ITiCSE '11.

[3]  Kathi Fisler,et al.  Measuring the effectiveness of error messages designed for novice programmers , 2011, SIGCSE.

[4]  A. J. Ko,et al.  Attitudes and self-efficacy in young adults' computing autobiographies , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[5]  Emerson R. Murphy-Hill,et al.  From Quick Fixes to Slow Fixes: Reimagining Static Analysis Resolutions to Enable Design Space Exploration , 2016, 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[6]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[7]  Maria Kutar,et al.  Cognitive Dimensions of Notations: Design Tools for Cognitive Technology , 2001, Cognitive Technology.

[8]  Matthias Felleisen,et al.  DrScheme: A Pedagogic Programming Environment for Scheme , 1997, PLILP.

[9]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[10]  Kathi Fisler,et al.  Mind your language: on novices' interactions with error messages , 2011, Onward! 2011.

[11]  Elliot Soloway,et al.  Learning to program = learning to construct mechanisms and explanations , 1986, CACM.

[12]  James J. Horning,et al.  What the Compiler Should Tell the User , 1976, Compiler Construction.

[13]  Kathi Fisler,et al.  The recurring rainfall problem , 2014, ICER '14.

[14]  John Maloney,et al.  The Scratch Programming Language and Environment , 2010, TOCE.

[15]  Ben Shneiderman,et al.  Designing computer system messages , 1982, CACM.

[16]  Benedict du Boulay,et al.  Fatal Error in Pass Zero: How not to Confuse Novices , 1984, Cognitive Ergonomics.

[17]  Michael Khoo,et al.  Do you care if a computer says sorry?: user experience design through affective messages , 2012, DIS '12.

[18]  Emerson R. Murphy-Hill,et al.  Compiler error notifications revisited: an interaction-first approach for helping developers more effectively comprehend and resolve error notifications , 2014, ICSE Companion.

[19]  Scott R. Klemmer,et al.  What would other programmers do: suggesting solutions to error messages , 2010, CHI.

[20]  John R. Anderson,et al.  Dynamic Student Modelling in an Intelligent Tutor for LISP Programming , 1985, IJCAI.

[21]  V. Javier Traver,et al.  On Compiler Error Messages: What They Say and What They Mean , 2010, Adv. Hum. Comput. Interact..

[22]  Bertrand Meyer,et al.  Compiler error messages: what can help novices? , 2008, SIGCSE '08.

[23]  Michael S. Horn,et al.  Tangible programming in the classroom with tern , 2007, CHI Extended Abstracts.

[24]  Titus Barik,et al.  Improving error notification comprehension through visual overlays in IDEs , 2014, 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[25]  Frederick W. B. Li,et al.  BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair , 2012, ICWL.

[26]  Thomas R. Flowers,et al.  Empowering students and building confidence in novice programmers through Gauntlet , 2004, 34th Annual Frontiers in Education, 2004. FIE 2004..