An application-independent system for visualizing user operation history

A history-of-user-operations function helps make applications easier to use. For example, users may have access to an operation history list in an application to undo or redo a past operation. To provide an overview of a long operation history and help users find target interactions or application states quickly, visual representations of operation history have been proposed. However, most previous systems are tightly integrated with target applications and difficult to apply to new applications. We propose an application-independent method that can visualize the operation history of arbitrary GUI applications by monitoring the input and output GUI events from outside of the target application. We implemented a prototype system that visualizes operation sequences of generic Java Awt/Swing applications using an annotated comic strip metaphor. We tested the system with various applications and present results from a user study.

[1]  Eser Kandogan,et al.  Koala: capture, share, automate, personalize business processes on the web , 2007, CHI.

[2]  Takeo Igarashi,et al.  Flatland: new dimensions in office whiteboards , 1999, CHI '99.

[3]  Victor Kaptelinin,et al.  Transient visual cues for scrolling: an empirical study , 2002, CHI Extended Abstracts.

[4]  Satoshi Matsuoka,et al.  Teddy: A Sketching Interface for 3D Freeform Design , 1999, SIGGRAPH Courses.

[5]  Henry Lieberman,et al.  Mondrian: a teachable graphical editor , 1993, INTERCHI.

[6]  Daniel Cohen-Or,et al.  Action synopsis: pose selection and illustration , 2005, ACM Trans. Graph..

[7]  Stefan Schlechtweg,et al.  Depicting Motion in Motionless Pictures , 1999 .

[8]  Patrick Baudisch,et al.  High-Density Cursor: a Visualization Technique that Helps Users Keep Track of Fast-moving Mouse Cursors , 2003, INTERACT.

[9]  Irfan A. Essa,et al.  Video-based nonphotorealistic and expressive illustration of motion , 2005, International 2005 Computer Graphics.

[10]  S. Su.  Visualizing , Editing , and Inferring Structure in 2 D Graphics , 2022 .

[11]  Michal Irani,et al.  Video indexing based on mosaic representations , 1998, Proc. IEEE.

[12]  Pat Hanrahan,et al.  Designing effective step-by-step assembly instructions , 2003, ACM Trans. Graph..

[13]  Tessa A. Lau,et al.  DocWizards: a system for authoring follow-me documentation wizards , 2005, UIST.

[14]  Yukinobu Taniguchi,et al.  PanoramaExcerpts: extracting and packing panoramas for video browsing , 1997, MULTIMEDIA '97.

[15]  Walter Bender,et al.  Salient video stills: content and context preserved , 1993, MULTIMEDIA '93.

[16]  Desney S. Tan,et al.  Phosphor: explaining transitions in the user interface using afterglow effects , 2006, UIST.

[17]  David Salesin,et al.  Schematic storyboarding for video visualization and editing , 2006, SIGGRAPH 2006.

[18]  Steven K. Feiner,et al.  A history-based macro by example system , 1992, UIST '92.

[19]  Kunio Kondo,et al.  Cartoon Blur: Non-Photorealistic Motion Blur. , 2003 .

[20]  Jun Rekimoto,et al.  Time-machine computing: a time-centric approach for the information environment , 1999, UIST '99.

[21]  Walter Bender,et al.  Salient Stills: Process and Practice , 1996, IBM Syst. J..

[22]  Chris Stolte,et al.  Rendering effective route maps: improving usability through generalization , 2001, SIGGRAPH.

[23]  Rob Miller,et al.  The Amulet Environment: New Models for Effective User Interface Software Development , 1997, IEEE Trans. Software Eng..

[24]  Maic Masuch,et al.  Speedlines: depicting motion in motionless pictures , 1999, SIGGRAPH '99.

[25]  Morgan Dixon,et al.  ExperiScope: an analysis tool for interaction data , 2007, CHI.

[26]  Xiaoyang Mao,et al.  Visualizing histories for selective undo and redo , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[27]  Pierre Dragicevic,et al.  Mnemonic rendering: an image-based approach for exposing hidden changes in dynamic displays , 2006, UIST.

[28]  Thomas Berlage,et al.  A selective undo mechanism for graphical user interfaces based on command objects , 1994, TCHI.

[29]  William T. Freeman,et al.  Shape-time photography , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[30]  Brad A. Myers,et al.  Pursuit: graphically representing programs in a demonstrational visual shell , 1994, CHI Conference Companion.