vsInk: An Extensible Framework for Adding Digital Ink to Visual Studio

This project adds digital ink annotations into the active code editor in Visual Studio 2010. Previous attempts had failed because of limitations in the older versions of Visual Studio and Eclipse. Visual Studio 2010 provides an extended API that allows direct modification of the active code editor window. This project uses the new API to add annotations and investigates several technical challenges with combining ink and text. These include:  Positioning strokes correctly when the document scrolled or changed  Handling changes to the underlying code  Handling collapsible regions  Grouping strokes into annotations vsInk partially solves all of these challenges. A user can add strokes to a code document. As strokes are added they are grouped together into annotations and anchored in the code. As code is changed (either by scrolling or editing) the annotations are repositioned. vsInk automatically hides annotations that are outside the editor view on in collapsed regions. The usability study shows vsInk solves the challenges but only partially. There are two main issues: strokes are incorrectly grouped into annotations and tall annotations disappear unexpectedly. Despite these issues the subjects found vsInk easy and enjoyable to use. vsInk opens options for future research into combining digital ink and code together and it provides a platform that can easily be extended.

[1]  Christof Lutteroth,et al.  Proceedings of the Eleventh Australasian Conference on User Interface - Volume 106 , 2010 .

[2]  Isabelle Guyon,et al.  UNIPEN project of on-line data exchange and recognizer benchmarks , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3 - Conference C: Signal Processing (Cat. No.94CH3440-5).

[3]  Chunyuan Liao,et al.  PapierCraft: A System for Interactive Paper , 2005 .

[4]  Peter J. Nürnberg,et al.  Proceedings of the Fifth ACM Conference on Digital Libraries, June 2-7, 2000, San Antonio, TX, USA , 2000 .

[5]  Jessica R. Cauchard.  Mobile multi-display environments , 2011, UIST '11 Adjunct.

[6]  Meredith Ringel Morris,et al.  Reading Revisited: Evaluating the Usability of Digital Display Surfaces for Active Reading Tasks , 2007, Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP'07).

[7]  Ivan E. Sutherland,et al.  Sketch pad a man-machine graphical communication system , 1964, DAC.

[8]  Nicola Ferro,et al.  A historical and contemporary study on annotations to derive key features for systems design , 2007, International Journal on Digital Libraries.

[9]  Richard Turner,et al.  The myth of the paperless office , 2001 .

[10]  Richard J. Anderson,et al.  Experiences with a tablet PC based lecture presentation system in computer science courses , 2004, SIGCSE '04.

[11]  Max Mühlhäuser,et al.  CoScribe: Using Paper for Collaborative Annotations in Lectures , 2008, 2008 Eighth IEEE International Conference on Advanced Learning Technologies.

[12]  Beryl Plimmer,et al.  RATA.Gesture: A gesture recognizer developed using data mining , 2012, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[13]  Paul A. Viola,et al.  Spatial recognition and grouping of text and graphics , 2004, SBM'04.

[14]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[15]  Abigail Sellen,et al.  The myth of the paperless office , 2001 .

[16]  David Bargeron,et al.  Robust annotation positioning in digital documents , 2001, CHI.

[17]  Bill N. Schilit,et al.  Dynomite: a dynamically organized ink and audio notebook , 1997, CHI.

[18]  Ralf Klamma,et al.  u-Annotate: An Application for User-Driven Freeform Digital Ink Annotation of E-Learning Content , 2006, Sixth IEEE International Conference on Advanced Learning Technologies (ICALT'06).

[19]  Ramanujan S. Kashi,et al.  An architecture for ink annotations on Web documents , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[20]  C.C. Marshall,et al.  Exploring the relationship between personal and public annotations , 2004, Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004..

[21]  Mark S. Ackerman,et al.  Successful classroom deployment of a social document annotation system , 2012, CHI.

[22]  David Bargeron,et al.  Reflowing digital ink annotations , 2003, CHI '03.

[23]  Beryl Plimmer,et al.  RCA: experiences with an IDE annotation tool , 2006, CHINZ '06.

[24]  Kenton O'Hara,et al.  A comparison of reading paper and on-line documents , 1997, CHI.

[25]  Maneesh Agrawala,et al.  DIZI: A Digital Ink Zooming Interface for Document Annotation , 2005, INTERACT.

[26]  B. Simon,et al.  Use of classroom presenter in engineering courses , 2005, Proceedings Frontiers in Education 35th Annual Conference.

[27]  Robert Wilensky,et al.  Robust intra-document locations , 2000, Comput. Networks.

[28]  Jerry Alan Fails,et al.  ScreenCrayons: annotating anything , 2004, UIST '04.

[29]  Bill N. Schilit,et al.  Linking by inking: trailblazing in a paper-like hypertext , 1998, HYPERTEXT '98.

[30]  Catherine C. Marshall,et al.  Introducing a digital library reading appliance into a reading group , 1999, DL '99.

[31]  Edward A. Fox,et al.  Proceedings of the Fourth ACM conference on Digital Libraries, August 11-14, 1999, Berkeley, CA, USA , 1999 .

[32]  Joanna L. Wolfe,et al.  Effects of annotations on student readers and writers , 2000, DL '00.

[33]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[34]  David Jones,et al.  Discerning structure from freeform handwritten notes , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[35]  Beryl Plimmer,et al.  Issues of extending the user interface of integrated development environments , 2008, CHINZ.

[36]  Catherine C. Marshall,et al.  Designing e-books for legal research , 2001, JCDL '01.

[37]  Andreas Dengel,et al.  Document Analysis Systems VI , 2004, Lecture Notes in Computer Science.

[38]  Bill N. Schilit,et al.  Beyond paper: supporting active reading with free form digital ink annotations , 1998, CHI.

[39]  Peter Brandl,et al.  NiCEBook: supporting natural note taking , 2010, CHI.

[40]  Catherine C. Marshall,et al.  Annotation: from paper books to the digital library , 1997, DL '97.

[41]  Xin Wang,et al.  Parsing ink annotations on heterogeneous documents , 2006, SBM'06.

[42]  Zile Wei,et al.  Recognizing Freeform Digital Ink Annotations , 2004, Document Analysis Systems.

[43]  Xin Wang,et al.  Ink Annotations and their Anchoring in Heterogeneous Digital Documents , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[44]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[45]  Beryl Plimmer,et al.  iAnnotate: Exploring Multi-User Ink Annotation in Web Browsers , 2010, AUIC.

[46]  Beryl Plimmer.  A Comparative Evaluation of Annotation Software for Grading Programming Assignments , 2010, AUIC.

[47]  Beryl Plimmer,et al.  A pen-based paperless environment for annotating and marking student assignments , 2006, AUIC.

[48]  Jan O. Borchers,et al.  CodeGraffiti: communication by sketching for pair programmers , 2010, UIST '10.

[49]  Richard J. Anderson,et al.  Classroom Presenter: Enhancing Interactive Education with Digital Ink , 2007, Computer.

[50]  B. Thomas,et al.  Usability Evaluation In Industry , 1996 .

[51]  Daniel T. Joyce,et al.  Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education, SIGCSE 2004, Norfolk, Virginia, USA, March 3-7, 2004 , 2004, SIGCSE.

[52]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.