Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study

ContextThere is a widespread belief in both SE and other branches of science that experience helps professionals to improve their performance. However, cases have been reported where experience not only does not have a positive influence but sometimes even degrades the performance of professionals.AimDetermine whether years of experience influence programmer performance.MethodWe have analysed 10 quasi-experiments executed both in academia with graduate and postgraduate students and in industry with professionals. The experimental task was to apply ITLD on two experimental problems and then measure external code quality and programmer productivity.ResultsProgramming experience gained in industry does not appear to have any effect whatsoever on quality and productivity. Overall programming experience gained in academia does tend to have a positive influence on programmer performance. These two findings may be related to the fact that, as opposed to deliberate practice, routine practice does not appear to lead to improved performance. Experience in the use of productivity tools, such as testing frameworks and IDE also has positive effects.ConclusionYears of experience are a poor predictor of programmer performance. Academic background and specialized knowledge of task-related aspects appear to be rather good predictors.

[1]  Natalia Juristo Juzgado,et al.  Evidence of the presence of bias in subjective metrics: analysis within a family of experiments , 2014, EASE '14.

[2]  R. O’Brien,et al.  A Caution Regarding Rules of Thumb for Variance Inflation Factors , 2007 .

[3]  Dorothea P. Simon,et al.  Expert and Novice Performance in Solving Physics Problems , 1980, Science.

[4]  Tore Dybå,et al.  Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise , 2007, IEEE Transactions on Software Engineering.

[5]  R. Hogarth,et al.  Research on Judgment and Decision Making: Currents, Connections, and Controversies , 1997 .

[6]  Kate Ehrlich,et al.  Empirical Studies of Programming Knowledge , 1984, IEEE Transactions on Software Engineering.

[7]  Ritu Agarwal,et al.  Knowledge extraction using content analysis , 1991 .

[8]  Kate Ehrlich,et al.  Cognitive strategies and looping constructs: an empirical study , 1983, CACM.

[9]  J. Reitman,et al.  Knowledge organization and skill differences in computer programmers , 1981, Cognitive Psychology.

[10]  B. Adelson.  Problem solving and the development of abstract categories in programming languages , 1981, Memory & cognition.

[11]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[12]  Leonard A. Jason,et al.  Handbook of Methodological Approaches to Community-Based Research: Qualitative, Quantitative, and Mixed Methods , 2015 .

[13]  James M. Bieman,et al.  Software Metrics: A Rigorous and Practical Approach, Third Edition , 2014 .

[14]  Michael C. Loui,et al.  Debugging: from novice to expert , 2004, SIGCSE '04.

[15]  Nancy J. Cooke,et al.  Effects of Computer Programming Experience on Network Representations of Abstract Programming Concepts , 1988, Int. J. Man Mach. Stud..

[16]  Simon P. Davies,et al.  Characterizing the program design activity : neither strictly top-down nor globally opportunistic , 1991 .

[17]  B. Adelson.  When Novices Surpass Experts: The Difficulty of a Task May Increase With Expertise , 1984 .

[18]  Magne Jørgensen,et al.  Characteristics of software engineers with optimistic predictions , 2007, J. Syst. Softw..

[19]  S. Green.  How Many Subjects Does It Take To Do A Regression Analysis. , 1991, Multivariate behavioral research.

[20]  John E. Hunter,et al.  Job experience correlates of job performance. , 1988 .

[21]  Richard M. Heiberger,et al.  Statistical Analysis and Data Display: An Intermediate Course with Examples in R , 2015 .

[22]  H. Simon,et al.  The mind's eye in chess. , 1973 .

[23]  Françoise Détienne,et al.  Object-Oriented Program Comprehension: Effect of Expertise, Task and Phase , 2002, Empirical Software Engineering.

[24]  Elliot Soloway,et al.  Studying the Novice Programmer , 1988 .

[25]  Ulman Lindenberger,et al.  Structural equation model trees. , 2013, Psychological methods.

[26]  Jean Scholtz,et al.  The Roles Beacons Play in Comprehension for Novice and Expert Programmers , 2002, PPIG.

[27]  F. Gobet,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[28]  Kenneth E. Iverson,et al.  A programming language , 1899, AIEE-IRE '62 (Spring).

[29]  C C ChanKeith,et al.  Pair programming productivity , 2006 .

[30]  Sven Apel,et al.  Measuring and modeling programming experience , 2013, Empirical Software Engineering.

[31]  Matthias M. Müller,et al.  An empirical study about the feelgood factor in pair programming , 2004, 10th International Symposium on Software Metrics, 2004. Proceedings..

[32]  S. Weisberg.  Applied Linear Regression: Weisberg/Applied Linear Regression 3e , 2005 .

[33]  Kristopher J Preacher,et al.  On the practice of dichotomization of quantitative variables. , 2002, Psychological methods.

[34]  S. Sonnentag.  Expertise in professional software design: a process study. , 1998, The Journal of applied psychology.

[35]  Paul J. Feltovich,et al.  An Introduction to Cambridge Handbook of Expertise and Expert Performance: Its Development, Organization, and Content , 2006 .

[36]  Susan Wiedenbeck,et al.  Novice/Expert Differences in Programming Skills , 1985, Int. J. Man Mach. Stud..

[37]  N. Charness,et al.  Expert Performance Its Structure and Acquisition , 2002 .

[38]  Marco Torchiano,et al.  The Role of Experience and Ability in Comprehension Tasks Supported by UML Stereotypes , 2007, 29th International Conference on Software Engineering (ICSE'07).

[39]  Emilia Mendes,et al.  Software productivity measurement using multiple size measures , 2004, IEEE Transactions on Software Engineering.

[40]  Lech Madeyski.  Preliminary Analysis of the Effects of Pair Programming and Test-Driven Development on the External Code Quality , 2005, Software Engineering: Evolution and Emerging Technologies.

[41]  David A. Belsley,et al.  Conditioning Diagnostics: Collinearity and Weak Data in Regression , 1991 .

[42]  Klaus Pohl,et al.  The impact of students' skills and experiences on empirical results: a controlled experiment with undergraduate and graduate students , 2015, EASE.

[43]  David P. Darcy,et al.  Exploring Individual Characteristics and Programming Performance: Implications for Programmer Selection , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[44]  Kimberly Chulis.  Optimal segmentation approach and application Clustering vs . classification trees , 2017 .

[45]  J. Miles,et al.  Applying regression & correlation : a guide for students and researchers , 2001 .

[46]  Lawrence G. Votta,et al.  By the way, has anyone studied any real programmers, yet? [software development process] , 1994, Proceedings. Ninth International Software Process Workshop.

[47]  Per Runeson,et al.  Using Students as Experiment Subjects – An Analysis on Graduate and Freshmen Student Data , 2003 .

[48]  Richard E. Mayer,et al.  From Novice to expert , 1988 .

[49]  K. A. Ericsson,et al.  Expert and exceptional performance: evidence of maximal adaptation to task constraints. , 1996, Annual review of psychology.

[50]  L. Pettit,et al.  Conditioning Diagnostics: Collinearity and Weak Data in Regression , 1992 .

[51]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[52]  Marco Torchiano,et al.  On the effectiveness of the test-first approach to programming , 2005, IEEE Transactions on Software Engineering.

[53]  Keith C. C. Chan,et al.  Pair programming productivity: Novice-novice vs. expert-expert , 2006, Int. J. Hum. Comput. Stud..

[54]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[55]  Matthias M. Müller,et al.  The effect of experience on the test-driven development process , 2007, Empirical Software Engineering.

[56]  R. Riley,et al.  Meta-analysis of individual participant data: rationale, conduct, and reporting , 2010, BMJ : British Medical Journal.

[57]  Joyce J. Elam,et al.  Semantic Structuring in Analyst Acquisition and Representation of Facts in Requirements Analysis , 1998, Inf. Syst. Res..

[58]  Bill Curtis,et al.  Modern Coding Practices and Programmer Performance , 1979, Computer.

[59]  John Elder,et al.  Handbook of Statistical Analysis and Data Mining Applications , 2009 .

[60]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[61]  Françoise Détienne,et al.  Mental Representations Constructed by Experts and Novices in Object-Oriented Program Comprehension , 1997, INTERACT.

[62]  Petek Askar,et al.  AN INVESTIGATION OF FACTORS RELATED TO SELF-EFFICACY FOR JAVA PROGRAMMING AMONG ENGINEERING STUDENTS , 2009 .

[63]  Lia Di Bello,et al.  Studying human expertise: Beyond the binary paradigm , 1996, J. Exp. Theor. Artif. Intell..

[64]  Christopher Winship,et al.  REGRESSION MODELS WITH ORDINAL VARIABLES , 1984 .

[65]  Yong Rae Kwon,et al.  Specification-based program slicing and its applications , 2001, J. Syst. Archit..

[66]  Phillip G. Armour.  Beware of counting LOC , 2004, CACM.

[67]  Roberta L. Klatzky,et al.  Expertise in a Computer Operating System: Conceptualization and Performance , 1990, Hum. Comput. Interact..

[68]  S. Sim.  An Initial Study to Develop an Empirical Test for Software Engineering Expertise , 2006 .

[69]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[70]  Himalaya Patel,et al.  An Improved Usability Measure Based on Novice and Expert Performance , 2011, Int. J. Hum. Comput. Interact..

[71]  Richard E. Mayer,et al.  Chapter 33 – From Novice to expert , 1997 .

[72]  A. D. D. Groot.  Thought and Choice in Chess , 1978 .

[73]  Bill Curtis,et al.  Fifteen years of psychology in software engineering: Individual differences and cognitive science , 1984, ICSE '84.

[74]  Kai Petersen,et al.  Considering rigor and relevance when evaluating test driven development: A systematic review , 2014, Inf. Softw. Technol..

[75]  Mark Weiser,et al.  Program Slicing , 1981, IEEE Transactions on Software Engineering.

[76]  G. Colvin,et al.  Talent Is Overrated: What Really Separates World-Class Performers from Everybody Else , 2008 .

[77]  Robin Jeffries,et al.  The Processes Involved in Designing Software. , 1980 .

[78]  Gavriel Salvendy,et al.  Quantitative and qualitative differences between experts and novices in chunking computer software knowledge , 1994, Int. J. Hum. Comput. Interact..

[79]  Colin Camerer,et al.  The process-performance paradox in expert judgment - How can experts know so much and predict so badly? , 1991 .

[80]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[81]  Laurie A. Williams,et al.  On the Effectiveness of Unit Test Automation at Microsoft , 2009, 2009 20th International Symposium on Software Reliability Engineering.

[82]  Robert J. Crutcher,et al.  The role of deliberate practice in the acquisition of expert performance. , 1993 .

[83]  Mark Weiser,et al.  Programming Problem Representation in Novice and Expert Programmers , 1983, Int. J. Man Mach. Stud..

[84]  Richard M. Heiberger,et al.  Statistical Analysis and Data Display: An Intermediate Course with Examples in S-Plus, R, and SAS , 2004 .

[85]  Bill Curtis,et al.  A field study of the software design process for large systems , 1988, CACM.

[86]  Sabine Sonnentag,et al.  Excellent software professionals: experience, work activities, and perception by peers , 1995, Behav. Inf. Technol..

[87]  K. A. Ericsson,et al.  The Influence of Experience and Deliberate Practice on the Development of Superior Expert Performance , 2006 .