Measuring programming experience

Programming experience is an important confounding parameter in controlled experiments regarding program comprehension. In literature, ways to measure or control programming experience vary. Often, researchers neglect it or do not specify how they controlled it. We set out to find a well-defined understanding of programming experience and a way to measure it. From published comprehension experiments, we extracted questions that assess programming experience. In a controlled experiment, we compare the answers of 128 students to these questions with their performance in solving program-comprehension tasks. We found that self estimation seems to be a reliable way to measure programming experience. Furthermore, we applied exploratory factor analysis to extract a model of programming experience. With our analysis, we initiate a path toward measuring programming experience with a valid and reliable tool, so that we can control its influence on program comprehension.

[1]  R. Peterson.  Constructing Effective Questionnaires , 1999 .

[2]  Matthias M. Müller,et al.  Are Reviews an Alternative to Pair Programming? , 2004, Empirical Software Engineering.

[3]  Raimund Dachselt,et al.  Using background colors to support program comprehension in software product lines , 2011, EASE.

[4]  Anneliese Amschler Andrews,et al.  Program Comprehension During Software Maintenance and Evolution , 1995, Computer.

[5]  Dietmar Pfahl,et al.  Reporting Experiments in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[6]  Peter M. Bentler,et al.  Practical Issues in Structural Modeling , 1987 .

[7]  Simon,et al.  Mental models, consistency and programming aptitude , 2008, ACE '08.

[8]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[9]  Sebastian Kleinschmager,et al.  How to rate programming skills in programming experiments?: a preliminary, exploratory, study based on university marks, pretests, and self-estimation , 2011, PLATEAU '11.

[10]  Herman Rubin,et al.  Statistical Inference in Factor Analysis , 1956 .

[11]  Sebastian Kleinschmager,et al.  Does aspect-oriented programming increase the development speed for crosscutting code? An empirical study , 2009, ESEM 2009.

[12]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[13]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[14]  Gail C. Murphy,et al.  Asking and Answering Questions during a Programming Change Task , 2008, IEEE Transactions on Software Engineering.

[15]  Tore Dybå,et al.  Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise , 2007, IEEE Transactions on Software Engineering.

[16]  Andreas Hasselberg,et al.  PROPHET : Tool Infrastructure To Support Program Comprehension Experiments , 2011 .

[17]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[18]  Marco Torchiano,et al.  On the effectiveness of the test-first approach to programming , 2005, IEEE Transactions on Software Engineering.

[19]  C MurphyGail,et al.  Asking and Answering Questions during a Programming Change Task , 2008 .

[20]  Mark Carpenter,et al.  The New Statistical Analysis of Data , 2000, Technometrics.

[21]  Dag I. K. Sjøberg,et al.  Effects of Personality on Pair Programming , 2010, IEEE Transactions on Software Engineering.

[22]  Walter F. Tichy,et al.  Hints for Reviewing Empirical Work in Software Engineering , 2000, Empirical Software Engineering.

[23]  Cláudio Sant'Anna,et al.  Evolving software product lines with aspects , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[24]  Stefan Biffl,et al.  Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[25]  Jason W. Osborne,et al.  Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. , 2005 .

[26]  M. Lewis-Beck.  Applied Regression: An Introduction , 1980 .

[27]  J. Yellott.  Correction for fast guessing and the speed-accuracy tradeoff in choice reaction time , 1971 .

[28]  S. Visalakshi,et al.  Shift in interests towards biotechnology in Indian pharmaceutical industry: an analysis , 2001 .

[29]  Marco Torchiano,et al.  The Role of Experience and Ability in Comprehension Tasks Supported by UML Stereotypes , 2007, 29th International Conference on Software Engineering (ICSE'07).

[30]  Christian Bunse,et al.  Using patterns for the refinement and translationof UML models: A controlled experiment , 2006, Empirical Software Engineering.