What is the foundation of evidence of human factors decisions in language design? an empirical study on programming language workshops

In recent years, the programming language design community has engaged in rigorous debate on the role of empirical evidence in the design of general purpose programming languages. Some scholars contend that the language community has failed to embrace a form of evidence that is non-controversial in other disciplines (e.g., medicine, biology, psychology, sociology, physics, chemistry), while others argue that a science of language design is unrealistic. While the discussion will likely persist for some time, we begin here a systematic evaluation of the use of empirical evidence with human users, documenting, paper-by-paper, the evidence provided for human factors decisions, beginning with 359 papers from the workshops PPIG, Plateau, and ESP. This preliminary work provides the following contributions: an analysis of the 1) overall quantity and quality of empirical evidence used in the workshops, and of the 2) overall significant challenges to reliably coding academic papers. We hope that, once complete, this long-term research project will serve as a practical catalog designers can use when evaluating the impact of a language feature on human users.

[1]  Nancy Pennington,et al.  Comprehension strategies in programming , 1987 .

[2]  Anneliese Amschler Andrews,et al.  Comprehension processes during large scale maintenance , 1994, Proceedings of 16th International Conference on Software Engineering.

[3]  Stefan Hanenberg,et al.  Faith, hope, and love: an essay on software science's neglect of human factors , 2010, OOPSLA.

[4]  Judith Good,et al.  Class participation and shyness: affect and learning to program , 2011, PPIG.

[5]  T. Kaptchuk,et al.  Intentional Ignorance: A History of Blind Assessment and Placebo Controls in Medicine , 1998, Bulletin of the history of medicine.

[6]  Margaret M. Burnett,et al.  A practical guide to controlled experiments of software engineering tools with human participants , 2013, Empirical Software Engineering.

[7]  Lutz Prechelt,et al.  An Empirical Comparison of Seven Programming Languages , 2000, Computer.

[8]  Westley Weimer,et al.  Benefits and barriers of user evaluation in software engineering research , 2011, OOPSLA '11.

[9]  W. Tichy.  Should Computer Scientists Experiment More? Computer Scientists and Practitioners Defend Their Lack of Experimentation with a Wide Range of Arguments. Some Arguments Suggest That , 1998 .

[10]  Richard J. Enbody,et al.  Python CS1 as preparation for C++ CS2 , 2009, SIGCSE '09.

[11]  Steve Cooper,et al.  Mediated transfer: Alice 3 to Java , 2012, SIGCSE '12.

[12]  Walter F. Tichy,et al.  Should Computer Scientists Experiment More? , 1998, Computer.

[13]  Brad A. Myers,et al.  Finding causes of program output with the Java Whyline , 2009, CHI.

[14]  Mary Shaw,et al.  The state of the art in end-user software engineering , 2011, ACM Comput. Surv..

[15]  Barbara Means,et al.  Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies , 2009 .

[16]  Caitlin Kelleher,et al.  Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers , 2005, CSUR.

[17]  N. Pennington.  Stimulus structures and mental representations in expert comprehension of computer programs , 1987, Cognitive Psychology.

[18]  Andreas Stefik,et al.  An Empirical Investigation into Programming Language Syntax , 2013, TOCE.

[19]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[20]  Steven Clarke.  Evaluating a new programming language , 2001, PPIG.

[21]  Christopher D. Hundhausen,et al.  Can direct manipulation lower the barriers to computer programming and promote transfer of training?: An experimental study , 2009, TCHI.

[22]  Shane Markstrum.  Staking claims: a history of programming language design claims and evidence: a positional work in progress , 2010, PLATEAU '10.

[23]  Stefan Hanenberg,et al.  Is Aspect-Oriented Programming a Rewarding Investment into Future Code Changes? A Socio-technical Study on Development and Maintenance Time , 2011, 2011 IEEE 19th International Conference on Program Comprehension.

[24]  Eric Rosenbaum,et al.  Scratch: programming for all , 2009, Commun. ACM.