ScreenCrayons: Using Screen Captures for Annotation and Research

SCREENCRAYONS: USING SCREEN CAPTURES FOR ANNOTATION AND RESEARCH Trent Taufer Department of Computer Science Master of Science In a world full of digital information we should be able to easily collect, organize, annotate, and leverage information from many different sources. This should be easy to do and not interrupt our normal workflow. A system to support information collection and organization should be user-friendly and as unobtrusive as possible, while still allowing for flexible and intelligent annotation. It should also be able to leverage the inherent information content of a collection of annotated information. We present a system that will demonstrate how these ideas can come together to make information collection easier and more productive. The system facilitates collection, organization, and annotation of information using screen captures, and leverages the information content of the annotated collection to automatically summarize the information and find additional related information via searchable document repositories. ScreenCrayons is a system for collecting annotations on any type of document or visual information from any application. The basis for the system is a screen capture upon which the user can highlight the relevant portions of the image. The user can define any number of topics for organizing notes. Each topic is associated with a highlighting "crayon." In addition the user can supply annotations in digital ink or text. Algorithms are described that summarize captured images based on the highlight strokes so as to provide overviews of many annotations as well as being able to "zoom in" on particular information about a given note and the context of that note. Annotations the user makes on the screen captures are automatically associated with regions of text that are then used to formulate queries to a search engine. The results of these queries are filtered and ranked based on their similarity to the original annotations. The system then presents links to these documents to the user. We also describe an experiment that shows that the documents found by the annotation system are generally found to be more relevant to a user's topic than the users own queries using the search engine.

[1]  Kenton O'Hara,et al.  A diary study of work-related reading: design implications for digital reading devices , 1998, CHI.

[2]  James P. Callan,et al.  Passage-level evidence in document retrieval , 1994, SIGIR '94.

[3]  Stefano Mizzaro,et al.  Evaluating user interfaces to information retrieval systems: a case study on user support , 1996, SIGIR '96.

[4]  Catherine C. Marshall,et al.  Designing e-books for legal research , 2001, JCDL '01.

[5]  Ramana Rao,et al.  Rich interaction in the digital library , 1995, CACM.

[6]  John Tait,et al.  Search strategies in content-based image retrieval , 2003, SIGIR.

[7]  Stuart K. Card,et al.  Degree-of-interest trees: a component of an attention-reactive user interface , 2002, AVI '02.

[8]  Bay-Wei Chang,et al.  Fluid visualization of spreadsheet structures , 1998, Proceedings. 1998 IEEE Symposium on Visual Languages (Cat. No.98TB100254).

[9]  Stuart K. Card,et al.  Information foraging in information access environments , 1995, CHI '95.

[10]  Jerry Alan Fails,et al.  A design tool for camera-based interaction , 2003, CHI '03.

[11]  Dan R. Olsen,et al.  Generalized pointing: enabling multiagent interaction , 1998, CHI.

[12]  Gerard Salton,et al.  Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer , 1989 .

[13]  Ellen Francik.  Rapid, integrated design of a multimedia communication system , 1995 .

[14]  Ramana Rao,et al.  Bridging the paper and electronic worlds: the paper user interface , 1993, INTERCHI.

[15]  Allison Woodruff,et al.  Enhancing a digital book with a reading recommender , 2000, CHI.

[16]  Jock D. Mackinlay,et al.  The document lens , 1993, UIST '93.

[17]  Bill N. Schilit,et al.  Dynomite: a dynamically organized ink and audio notebook , 1998 .

[18]  Ron Weiss,et al.  Fast and effective query refinement , 1997, SIGIR '97.

[19]  Stuart K. Card.  Pioneers and settlers: methods used in successful user interface design , 1995 .

[20]  Bill N. Schilit,et al.  Beyond paper: supporting active reading with free form digital ink annotations , 1998, CHI.

[21]  Stuart K. Card,et al.  Visualizing retrieved information: a survey , 1996, IEEE Computer Graphics and Applications.

[22]  Stuart K. Card,et al.  Information foraging models of browsers for very large document spaces , 1998, AVI '98.

[23]  Frank G. Halasz,et al.  Reflections on NoteCards: seven issues for the next generation of hypermedia systems , 1987, CACM.

[24]  Marc M. Sebrechts,et al.  Visualization of search results: a comparative evaluation of text, 2D, and 3D interfaces , 1999, SIGIR '99.

[25]  David Bargeron,et al.  Robust annotation positioning in digital documents , 2001, CHI.

[26]  John Tait,et al.  Evaluating a content based image retrieval system , 2001, SIGIR '01.

[27]  Jock D. Mackinlay,et al.  The structure of the information visualization design space , 1997, Proceedings of VIZ '97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium.

[28]  Michael I. Jordan,et al.  Modeling annotated data , 2003, SIGIR.

[29]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.

[30]  Bill N. Schilit,et al.  From reading to retrieval: freeform ink annotations as queries , 1999, SIGIR '99.

[31]  Laurent Denoue,et al.  Moving markup: repositioning freeform annotations , 2002, UIST '02.

[32]  Kristian J. Hammond,et al.  User interactions with everyday applications as context for just-in-time information access , 2000, IUI '00.

[33]  Thomas M. Breuel,et al.  Two Geometric Algorithms for Layout Analysis , 2002, Document Analysis Systems.

[34]  David Bargeron,et al.  Reflowing digital ink annotations , 2003, CHI '03.

[35]  Marcel Götze,et al.  The intelligent pen: toward a uniform treatment of electronic documents , 2002, SMARTGRAPH '02.