RichReview: blending ink, speech, and gesture to support collaborative document review

This paper introduces a novel document annotation system that aims to enable the kinds of rich communication that usually only occur in face-to-face meetings. Our system, RichReview, lets users create annotations on top of digital documents using three main modalities: freeform inking, voice for narration, and deictic gestures in support of voice. RichReview uses novel visual representations and time-synchronization between modalities to simplify annotation access and navigation. Moreover, RichReview's versatile support for multi-modal annotations enables users to mix and interweave different modalities in threaded conversations. A formative evaluation demonstrates early promise for the system finding support for voice, pointing, and the combination of both to be especially valuable. In addition, initial findings point to the ways in which both content and social context affect modality choice.

[1]  Ivan Poupyrev,et al.  Proceedings of the 26th annual ACM symposium on User interface software and technology , 2013, UIST 2013.

[2]  William Buxton,et al.  Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display , 2002, UIST '02.

[3]  Abigail Sellen,et al.  The myth of the paperless office , 2001 .

[4]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[5]  Colin J. Toms.  Write here, write now , 2000 .

[6]  Kellogg S. Booth,et al.  Co-authoring with structured annotations , 2006, CHI.

[7]  Aaron E. Rosenberg,et al.  SCANMail: a voicemail interface that makes speech browsable, readable and searchable , 2002, CHI.

[8]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[9]  W. Buxton,et al.  Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display , 2002, UIST '02.

[10]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[11]  Davida Charney,et al.  Distributed collaborative writing: a comparison of spoken and written modalities for reviewing and revising documents , 1994, CHI '94.

[12]  SYLVIE NOËL,et al.  Empirical Study on Collaborative Writing: What Do Co-authors Do, Use, and Like? , 2004, Computer Supported Cooperative Work (CSCW).

[13]  Jonathan Grudin,et al.  Why CSCW applications fail: problems in the design and evaluationof organizational interfaces , 1988, CSCW '88.

[14]  Meredith Ringel Morris,et al.  Reading Revisited: Evaluating the Usability of Digital Display Surfaces for Active Reading Tasks , 2007, Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP'07).

[15]  Robert E. Kraut,et al.  The use of visual information in shared visual spaces: informing the development of virtual co-presence , 2002, CSCW '02.

[16]  Timothy W. Bickmore,et al.  The Role of Gesture in Document Explanation by Embodied Conversational Agents , 2008, Int. J. Semantic Comput..

[17]  Carl Gutwin,et al.  Faster document navigation with space-filling thumbnails , 2006, CHI.

[18]  Barry Arons,et al.  The audio notebook: paper and pen interaction with structured speech , 2001, CHI.

[19]  Steve Whittaker,et al.  A preliminary analysis of the products of HCI research, using Pro Forma abstracts , 1994, CHI Conference Companion.

[20]  Robert E. Kraut,et al.  Expressive richness: a comparison of speech and text as media for revision , 1991, CHI.

[21]  Catherine C. Marshall,et al.  Collaborating over portable reading appliances , 2005, Personal Technologies.

[22]  Stephen R. Levine,et al.  The Freestyle System , 1991 .

[23]  D. McNeill.  Language and Gesture , 2000 .

[24]  Robert T. Nicholson,et al.  Usage patterns in an integrated voice and data communications system , 1985, TOIS.

[25]  C.C. Marshall,et al.  Exploring the relationship between personal and public annotations , 2004, Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004..

[26]  Robert E. Kraut,et al.  Task Requirements and Media Choice in Collaborative Writing , 1992, Hum. Comput. Interact..

[27]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[28]  William Buxton,et al.  A marking based interface for collaborative writing , 1993, UIST '93.

[29]  Jeremy P. Birnholtz,et al.  Write here, write now!: an experimental study of group maintenance in collaborative writing , 2013, CHI.

[30]  Bill N. Schilit,et al.  Dynomite: a dynamically organized ink and audio notebook , 1997, CHI.

[31]  Nicholas Chen,et al.  TextTearing: opening white space for digital ink annotation , 2013, UIST.