[Journal First] Measuring Program Comprehension: A Large-Scale Field Study with Professionals

This paper is published in IEEE Transaction on Software Engineering (DOI: 10.1109/TSE.2017.2734091). Comparing with previous programming comprehension studies that are usually in controlled settings or have a small number of participants, we perform a more realistic investigation of program comprehension activities. To do this, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We collect 3,148 working hour data from 78 professional developers in a field study. We follow Minelli et al.'s approach to assign developers' activities into four categories: navigation, editing, comprehension, and other. Then we measure comprehension time by calculating the time that developers spend on program comprehension. We find that on average developers spend ~58% of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers' experience, and project phase on the time that is spent on program comprehension.

[1]  Lori L. Pollock,et al.  Automatic generation of natural language summaries for Java classes , 2013, 2013 21st International Conference on Program Comprehension (ICPC).

[2]  Brad A. Myers,et al.  An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks , 2006, IEEE Transactions on Software Engineering.

[3]  Zhenchang Xing,et al.  ActivitySpace: A Remembrance Framework to Support Interapplication Information Needs , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[4]  Emerson R. Murphy-Hill,et al.  Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development? , 2014, ICSE.

[5]  Brad A. Myers,et al.  Designing the whyline: a debugging interface for asking questions about program behavior , 2004, CHI.

[6]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[7]  Tarun Jaiswal,et al.  Fundamentals of Software Engineering , 2017, Lecture Notes in Computer Science.

[8]  Zhenchang Xing,et al.  Measuring Program Comprehension: A Large-Scale Field Study with Professionals , 2018, IEEE Transactions on Software Engineering.

[9]  Michele Lanza,et al.  I know what you did last summer: an investigation of how developers spend their time , 2015, ICPC '15.

[10]  Jonathan I. Maletic,et al.  An Eye Tracking Study on camelCase and under_score Identifier Styles , 2010, 2010 IEEE 18th International Conference on Program Comprehension.

[11]  Seung-won Hwang,et al.  Enriching Documents with Examples: A Corpus Mining Approach , 2013, TOIS.

[12]  Gail C. Murphy,et al.  Impact of developer turnover on quality in open-source software , 2015, ESEC/SIGSOFT FSE.

[13]  Audris Mockus,et al.  Quantifying and Mitigating Turnover-Induced Knowledge Loss: Case Studies of Chrome and a Project at Avaya , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[14]  Jing Li,et al.  Extracting and analyzing time-series HCI data from screen-captured task videos , 2016, Empirical Software Engineering.

[15]  Janice Singer,et al.  NavTracks: supporting navigation in software , 2005, 13th International Workshop on Program Comprehension (IWPC'05).

[16]  Gary Klein,et al.  Supervisor Support and Career Anchor Impact on the Career Satisfaction of the Entry-Level Information Systems Professional , 1999, J. Manag. Inf. Syst..

[17]  Sushil Krishna Bajracharya,et al.  CodeGenie: using test-cases to search and reuse source code , 2007, ASE '07.

[18]  Diana Adler,et al.  Using Multivariate Statistics , 2016 .

[19]  Anneliese Amschler Andrews,et al.  Program Comprehension During Software Maintenance and Evolution , 1995, Computer.

[20]  Rainer Koschke,et al.  How do professional developers comprehend software? , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[21]  David Lo,et al.  How practitioners perceive the relevance of software engineering research , 2015, ESEC/SIGSOFT FSE.

[22]  Xiaochen Li,et al.  Query Expansion Based on Crowd Knowledge for Code Search , 2016, IEEE Transactions on Services Computing.

[23]  Agile Manifesto,et al.  Manifesto for Agile Software Development , 2001 .

[24]  Marvin V. Zelkowitz,et al.  Principles of software engineering and design , 1979 .

[25]  Joseph J. LaViola,et al.  Code bubbles: rethinking the user interface paradigm of integrated development environments , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[26]  Jeffrey C. Carver,et al.  How Practitioners Perceive the Relevance of ESEM Research , 2016, ESEM.

[27]  J. H. Zar,et al.  Significance Testing of the Spearman Rank Correlation Coefficient , 1972 .

[28]  Collin McMillan,et al.  Automatic documentation generation via source code summarization of method context , 2014, ICPC 2014.

[29]  Paul A. Watters,et al.  Statistics in a nutshell , 2008 .

[30]  Gabriele Bavota,et al.  Automatic query reformulations for text retrieval in software engineering , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[31]  Sven Apel,et al.  Measuring and modeling programming experience , 2013, Empirical Software Engineering.

[32]  Arthur J. Riel,et al.  Object-Oriented Design Heuristics , 1996 .

[33]  Thomas A. Corbi,et al.  Program Understanding: Challenge for the 1990s , 1989, IBM Syst. J..

[34]  Rainer Koschke,et al.  On the Comprehension of Program Comprehension , 2014, TSEM.

[35]  Andrea De Lucia,et al.  Using IR methods for labeling source code artifacts: Is it worthwhile? , 2012, 2012 20th IEEE International Conference on Program Comprehension (ICPC).

[36]  Frank Maurer,et al.  Requirements engineering and agile software development , 2003, WET ICE 2003. Proceedings. Twelfth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2003..

[37]  S. Shapiro,et al.  An analysis of variance test for normality ( complete samp 1 es ) t , 2007 .

[38]  D. J. Cook,et al.  4th international conference on software engineering , 1979 .

[39]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[40]  Jinqiu Yang,et al.  AutoComment: Mining question and answer sites for automatic comment generation , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[41]  Zhenchang Xing,et al.  Tracking and Analyzing Cross-Cutting Activities in Developers' Daily Work (N) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[42]  Mik Kersten,et al.  Using task context to improve programmer productivity , 2006, SIGSOFT '06/FSE-14.

[43]  Sushil Krishna Bajracharya,et al.  Sourcerer: a search engine for open source code supporting structure-based search , 2006, OOPSLA '06.

[44]  Sushil Krishna Bajracharya,et al.  Sourcerer: mining and searching internet-scale software repositories , 2008, Data Mining and Knowledge Discovery.

[45]  Barbee Teasley,et al.  The effects of naming style and expertise on program comprehension , 1994, Int. J. Hum. Comput. Stud..

[46]  Audris Mockus,et al.  Succession: Measuring transfer of code and developer productivity , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[47]  Chanchal Kumar Roy,et al.  Towards a context-aware IDE-based meta search engine for recommendation about programming errors and exceptions , 2014, 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE).

[48]  Gerald M. Weinberg,et al.  Psychology of computer programming , 1971 .

[49]  Arie van Deursen,et al.  Test confessions: A study of testing practices for plug-in systems , 2011, 2012 34th International Conference on Software Engineering (ICSE).

[50]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[51]  Ahmed E. Hassan,et al.  On the relationship between comment update practices and Software Bugs , 2012, J. Syst. Softw..

[52]  H. Pashler.  Dual-task interference in simple tasks: data and theory. , 1994, Psychological bulletin.

[53]  Mauricio A. Saca.  Refactoring improving the design of existing code , 2017, 2017 IEEE 37th Central America and Panama Convention (CONCAPAN XXXVII).

[54]  Michele Lanza,et al.  Seahawk: Stack Overflow in the IDE , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[55]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[56]  Thomas D. LaToza,et al.  Maintaining mental models: a study of developer work habits , 2006, ICSE.

[57]  Emily Hill,et al.  Towards automatically generating summary comments for Java methods , 2010, ASE.

[58]  Lori L. Pollock,et al.  Automatically detecting and describing high level actions within methods , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[59]  David W. Binkley,et al.  To camelcase or under_score , 2009, 2009 IEEE 17th International Conference on Program Comprehension.

[60]  Stéphane Ducasse,et al.  Polymetric Views - A Lightweight Visual Approach to Reverse Engineering , 2003, IEEE Trans. Software Eng..