Codemotion: expanding the design space of learner interactions with computer programming tutorial videos

Love them or hate them, videos are a pervasive format for delivering online education at scale. They are especially popular for computer programming tutorials since videos convey expert narration alongside the dynamic effects of editing and running code. However, these screencast videos simply consist of raw pixels, so there is no way to interact with the code embedded inside of them. To expand the design space of learner interactions with programming videos, we developed Codemotion, a computer vision algorithm that automatically extracts source code and dynamic edits from existing videos. Codemotion segments a video into regions that likely contain code, performs OCR on those segments, recognizes source code, and merges together related code edits into contiguous intervals. We used Codemotion to build a novel video player and then elicited interaction design ideas from potential users by running an elicitation study with 10 students followed by four participatory design workshops with 12 additional students. Participants collectively generated ideas for 28 kinds of interactions such as inline code editing, code-based skimming, pop-up video search, and in-video coding exercises.

[1]  Juliet M. Corbin,et al.  Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory , 2008 .

[2]  Tovi Grossman,et al.  Waken: reverse engineering usage information and interface structure from software videos , 2012, UIST '12.

[3]  Tovi Grossman,et al.  Swifter: improved online video scrubbing , 2013, CHI.

[4]  R. Smith,et al.  An Overview of the Tesseract OCR Engine , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[5]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Ray Smith.  An Overview of the Tesseract OCR Engine , 2007 .

[7]  Mark Guzdial,et al.  Effective Discussion Through a Computer-Mediated Anchored Forum , 2000 .

[8]  Björn Hartmann,et al.  Video digests: a browsable, skimmable format for informational lecture videos , 2014, UIST.

[9]  Rob Miller,et al.  GUI testing using computer vision , 2010, CHI.

[10]  Tovi Grossman,et al.  Investigating the feasibility of extracting tool demonstrations from in-situ video content , 2014, CHI.

[11]  Jiri Matas,et al.  Robust Detection of Lines Using the Progressive Probabilistic Hough Transform , 2000, Comput. Vis. Image Underst..

[12]  Tovi Grossman,et al.  Video lens: rapid playback and exploration of large video collections and associated metadata , 2014, UIST.

[13]  Margaret-Anne D. Storey,et al.  Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube , 2015, 2015 IEEE 23rd International Conference on Program Comprehension.

[14]  Michael J. Muller,et al.  Participatory design , 1993, CACM.

[15]  Tessa A. Lau,et al.  DocWizards: a system for authoring follow-me documentation wizards , 2005, UIST.

[16]  Scott R. Klemmer,et al.  Example-centric programming: integrating web search into the development environment , 2010, CHI.

[17]  Davide Fucci,et al.  Find, understand, and extend development screencasts on YouTube , 2017, SWAN@ESEC/SIGSOFT FSE.

[18]  Morgan Dixon,et al.  Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure , 2010, CHI.

[19]  Pei-Yu Chi,et al.  MixT: automatic generation of step-by-step mixed media tutorials , 2012, UIST.

[20]  Philip J. Guo,et al.  How video production affects student engagement: an empirical study of MOOC videos , 2014, L@S.

[21]  Michelle K. Smith,et al.  Active learning increases student performance in science, engineering, and mathematics , 2014, Proceedings of the National Academy of Sciences.

[22]  Grant Williams,et al.  Analyzing User Comments on YouTube Coding Tutorial Videos , 2017, 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC).

[23]  Michael Twidale,et al.  Over the Shoulder Learning: Supporting Brief Informal Learning , 2005, Computer Supported Cooperative Work (CSCW).

[24]  Gabriele Bavota,et al.  Too Long; Didn't Watch! Extracting Relevant Fragments from Software Development Video Tutorials , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[25]  Shengdong Zhao,et al.  NoteVideo: facilitating navigation of blackboard-style lecture videos , 2013, CHI.

[26]  Edward Cutrell,et al.  VidWiki: enabling the crowd to improve the legibility of online educational videos , 2014, CSCW.

[27]  Krzysztof Z. Gajos,et al.  Crowdsourcing step-by-step information extraction to enhance existing how-to videos , 2014, CHI.

[28]  Yi Yang,et al.  Content-Based Video Search over 1 Million Videos with 1 Core in 1 Second , 2015, ICMR.

[29]  Mira Dontcheva,et al.  Pause-and-play: automatically linking screencast video tutorials with applications , 2011, UIST.

[30]  Bill Buxton,et al.  Sketching User Experiences: Getting the Design Right and the Right Design , 2007 .

[31]  Björn Hartmann,et al.  SceneSkim: Searching and Browsing Movies Using Synchronized Captions, Scripts and Plot Summaries , 2015, UIST.

[32]  Philip J. Guo.  Non-Native English Speakers Learning Computer Programming: Barriers, Desires, and Design Opportunities , 2018, CHI.

[33]  Pei-Yu Chi,et al.  MixT: automatic generation of step-by-step mixed media tutorials , 2012, CHI Extended Abstracts.

[34]  Frédo Durand,et al.  Visual transcripts , 2015, ACM Trans. Graph..

[35]  Mark S. Ackerman,et al.  Successful classroom deployment of a social document annotation system , 2012, CHI.

[36]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[37]  Takeo Igarashi,et al.  Generating photo manipulation tutorials by demonstration , 2009, ACM Trans. Graph..

[38]  Tovi Grossman,et al.  Chronicle: capture, exploration, and playback of document workflow histories , 2010, UIST.

[39]  Krzysztof Z. Gajos,et al.  Data-driven interaction techniques for improving navigation of educational videos , 2014, UIST.

[40]  Rebecca Krosnick,et al.  VideoDoc : combining videos and lecture notes for a better learning experience , 2015 .

[41]  Margaret-Anne D. Storey,et al.  Documenting and sharing software knowledge using screencasts , 2017, Empirical Software Engineering.

[42]  Eran Yahav,et al.  Extracting code from programming tutorial videos , 2016, Onward!.

[43]  N. Hoffart.  Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[44]  Rob Miller,et al.  Sikuli: using GUI screenshots for search and automation , 2009, UIST '09.

[45]  Daniel Jackson,et al.  Panopticon: a parallel video overview system , 2013, UIST.