Extracting and analyzing time-series HCI data from screen-captured task videos

Recent years have witnessed the increasing emphasis on human aspects in software engineering research and practices. Our survey of existing studies on human aspects in software engineering shows that screen-captured videos have been widely used to record developers’ behavior and study software engineering practices. The screen-captured videos provide direct information about which software tools the developers interact with and which content they access or generate during the task. Such Human-Computer Interaction (HCI) data can help researchers and practitioners understand and improve software engineering practices from human perspective. However, extracting time-series HCI data from screen-captured task videos requires manual transcribing and coding of videos, which is tedious and error-prone. In this paper we report a formative study to understand the challenges in manually transcribing screen-captured videos into time-series HCI data. We then present a computer-vision based video scraping technique to automatically extract time-series HCI data from screen-captured videos. We also present a case study of our scvRipper tool that implements the video scraping technique using 29-hours of task videos of 20 developers in two development tasks. The case study not only evaluates the runtime performance and robustness of the tool, but also performs a detailed quantitative analysis of the tool’s ability to extract time-series HCI data from screen-captured task videos. We also study the developer’s micro-level behavior patterns in software development from the quantitative analysis.

[1]  Richard A. Parker,et al.  Designing and Conducting Survey Research: A Comprehensive Guide , 1992 .

[2]  Bo Zhang,et al.  StatSnowball: a statistical approach to extracting entity relationships , 2009, WWW '09.

[3]  Rob Miller,et al.  GUI testing using computer vision , 2010, CHI.

[4]  Bruce Phillips,et al.  Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems , 2008, CHI.

[5]  Tovi Grossman,et al.  Waken: reverse engineering usage information and interface structure from software videos , 2012, UIST '12.

[6]  Anthony F. Norcio,et al.  The Effect of Human Memory Organization on Code Reviews under Different Single and Pair Code Reviewing Scenarios , 2005 .

[7]  Jing Li,et al.  scvRipper: Video Scraping Tool for Modeling Developers' Behavior Using Interaction Data , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[8]  Paul Hildreth,et al.  Communities of practice : creating learning environments for educators , 2008 .

[9]  Qi Zhang,et al.  SIFT implementation and optimization for multi-core systems , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[10]  Tovi Grossman,et al.  CommunityCommands: command recommendations for software applications , 2009, UIST '09.

[11]  Tovi Grossman,et al.  Sketch-sketch revolution: an engaging tutorial system for guided sketching and application learning , 2011, UIST.

[12]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[13]  Brad A. Myers,et al.  A framework and methodology for studying the causes of software errors in programming systems , 2005, J. Vis. Lang. Comput..

[14]  Emerson R. Murphy-Hill,et al.  Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development? , 2014, ICSE.

[15]  Wei-Ying Ma,et al.  Web object retrieval , 2007, WWW '07.

[16]  Tom Yeh,et al.  Associating the visual representation of user interfaces with their internal structures and metadata , 2011, UIST.

[17]  Philip J. Guo,et al.  Two studies of opportunistic programming: interleaving web foraging, learning, and writing code , 2009, CHI.

[18]  A. Bandura.  Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[19]  Scott R. Klemmer,et al.  What would other programmers do: suggesting solutions to error messages , 2010, CHI.

[20]  Takeo Igarashi,et al.  An application-independent system for visualizing user operation history , 2008, UIST '08.

[21]  Zhenchang Xing,et al.  Improving feature location practice with multi-faceted interactive exploration , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[22]  Maarten van Someren,et al.  The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes , 1994 .

[23]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[24]  Brad A. Myers,et al.  Design requirements for more flexible structured editors from a study of programmers' text editing , 2005, CHI Extended Abstracts.

[25]  Wei-Ying Ma,et al.  Object-level ranking: bringing order to Web objects , 2005, WWW '05.

[26]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[27]  L Corritorecynthia,et al.  An exploratory study of program comprehension strategies of procedural and object-oriented programmers , 2001 .

[28]  Ryen W. White,et al.  The search dashboard: how reflection and comparison impact search behavior , 2012, CHI.

[29]  David F. Redmiles,et al.  Extracting usability information from user interface events , 2000, CSUR.

[30]  Susan Wiedenbeck,et al.  Direction and scope of comprehension-related activities by procedural and object-oriented programmers: an empirical study , 2000, Proceedings IWPC 2000. 8th International Workshop on Program Comprehension.

[31]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Marti A. Hearst,et al.  The state of the art in automating usability evaluation of user interfaces , 2001, CSUR.

[33]  Rachel K. E. Bellamy,et al.  Modeling programmer navigation: A head-to-head empirical evaluation of predictive models , 2011, 2011 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[34]  Jürgen Steimle,et al.  More than touch: understanding how people use skin as an input surface for mobile computing , 2014, CHI.

[35]  Martin P. Robillard,et al.  How effective developers investigate source code: an exploratory study , 2004, IEEE Transactions on Software Engineering.

[36]  W. Tsai,et al.  Spatial-domain image hiding using image differencing , 2000 .

[37]  James D. Herbsleb,et al.  Reading the documentation of invoked API functions in program comprehension , 2009, 2009 IEEE 17th International Conference on Program Comprehension.

[38]  Eleni Stroulia,et al.  From run-time behavior to usage scenarios: an interaction-pattern mining approach , 2002, KDD.

[39]  Thad Starner,et al.  Remembrance Agent: A Continuously Running Automated Information Retrieval System , 1996, PAAM.

[40]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[41]  Dina Katabi,et al.  Enabling Configuration-Independent Automation by Non-Expert Users , 2010, OSDI.

[42]  Christopher D. Hundhausen,et al.  A methodology for analyzing the temporal evolution of novice programs based on semantic components , 2006, ICER '06.

[43]  Yakup Genc,et al.  GPU-based Video Feature Tracking And Matching , 2006 .

[44]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[45]  Susan Wiedenbeck,et al.  An exploratory study of program comprehension strategies of procedural and object-oriented programmers , 2001, Int. J. Hum. Comput. Stud..

[46]  Tovi Grossman,et al.  Chronicle: capture, exploration, and playback of document workflow histories , 2010, UIST.

[47]  Brad A. Myers,et al.  An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks , 2006, IEEE Transactions on Software Engineering.

[48]  A. Collins,et al.  Situated Cognition and the Culture of Learning , 1989 .

[49]  Gregg Rothermel,et al.  How well do professional developers test with code coverage visualizations? An empirical study , 2005, 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'05).

[50]  Martin P. Robillard,et al.  Asking and answering questions about unfamiliar APIs: An exploratory study , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[51]  Brian D. Fisher,et al.  Managing software change tasks: an exploratory study , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[52]  Rafael C. González,et al.  Local Determination of a Moving Contrast Edge , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[54]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[55]  Michele Lanza,et al.  Seahawk: Stack Overflow in the IDE , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[56]  Dennis E. Egan,et al.  Handbook of Human Computer Interaction , 1988 .

[57]  Rachel K. E. Bellamy,et al.  How Programmers Debug, Revisited: An Information Foraging Theory Perspective , 2013, IEEE Transactions on Software Engineering.

[58]  Hongfang Liu,et al.  An investigation of the effect of module size on defect prediction using static measures , 2005, PROMISE@ICSE.

[59]  Anita Sarma,et al.  Tesseract: Interactive visual exploration of socio-technical relationships in software development , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[60]  Zhenchang Xing,et al.  What help do developers seek, when and how? , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[61]  Zhenchang Xing,et al.  Tracking and Analyzing Cross-Cutting Activities in Developers' Daily Work (N) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[62]  Mark Guzdial,et al.  Whorf: a visualization tool for software maintenance , 1992, Proceedings IEEE Workshop on Visual Languages.

[63]  Manolis Mavrikis,et al.  Exploring the Potential of Speech Recognition to Support Problem Solving and Reflection - Wizards Go to School in the Elementary Maths Classroom , 2014, EC-TEL.

[64]  A. Marie Vans,et al.  Program understanding behavior during debugging of large scale software , 1997, ESP '97.

[65]  Albert-László Barabási,et al.  The origin of bursts and heavy tails in human dynamics , 2005, Nature.

[66]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[67]  Gail C. Murphy,et al.  Fishtail: from task context to source code examples , 2011, TOPI '11.

[68]  Arie van Deursen,et al.  Test confessions: A study of testing practices for plug-in systems , 2011, 2012 34th International Conference on Software Engineering (ICSE).

[69]  Ramanathan V. Guha,et al.  Semantic search , 2003, WWW '03.

[70]  Morgan Dixon,et al.  Content and hierarchy in pixel-based methods for reverse engineering interface structure , 2011, CHI.

[71]  Amy J. Ko,et al.  Eliciting design requirements for maintenance-oriented IDEs: a detailed study of corrective and perfective maintenance tasks , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[72]  Jure Leskovec,et al.  Finding progression stages in time-evolving event sequences , 2014, WWW.

[73]  Thomas Fritz,et al.  Developers' code context models for change tasks , 2014, SIGSOFT FSE.

[74]  Tovi Grossman,et al.  Ambient help , 2011, CHI.

[75]  Tobias Ley,et al.  Open Learning and Teaching in Educational Communities , 2014, Lecture Notes in Computer Science.

[76]  Rachel K. E. Bellamy,et al.  Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks , 2008, CHI.

[77]  Scott E. Hudson,et al.  Automatically identifying targets users interact with during real world tasks , 2010, IUI '10.

[78]  Zhenchang Xing,et al.  An exploratory study of feature location process: Distinct phases, recurring patterns, and elementary actions , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[79]  Mark R. Leary,et al.  Introduction to behavioral research methods , 1990 .

[80]  F. Windmeijer,et al.  An R-squared measure of goodness of fit for some common nonlinear regression models , 1997 .

[81]  Rob Miller,et al.  Sikuli: using GUI screenshots for search and automation , 2009, UIST '09.

[82]  Stas Negara,et al.  Use, disuse, and misuse of automated refactorings , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[83]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[84]  Scott R. Klemmer,et al.  Example-centric programming: integrating web search into the development environment , 2010, CHI.

[85]  Gautam Shroff,et al.  Distributed side-by-side programming , 2009, 2009 ICSE Workshop on Cooperative and Human Aspects on Software Engineering.

[86]  Morgan Dixon,et al.  Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure , 2010, CHI.

[87]  Jesse H. Poore,et al.  Markov analysis of software specifications , 1993, TSEM.

[88]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[89]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[90]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[91]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[92]  Zhenchang Xing,et al.  ActivitySpace: A Remembrance Framework to Support Interapplication Information Needs , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[93]  Brad A. Myers,et al.  Designing the whyline: a debugging interface for asking questions about program behavior , 2004, CHI.

[94]  Marwan Abi-Antoun,et al.  Empirical Evaluation of Diagrams of the Run-time Structure for Coding Tasks , 2012, 2012 19th Working Conference on Reverse Engineering.