An optimization-based approach to dynamic visual context management

We are building an intelligent multimodal conversation system to aid users in exploring large and complex data sets. To tailor to diverse user queries introduced during a conversation, we automate the generation of system responses, including both spoken and visual outputs. In this paper, we focus on the problem of visual context management, a process that dynamically updates an existing visual display to effectively incorporate new information requested by subsequent user queries. Specifically, we develop an optimization based approach to visual context management. Compared to existing approaches, which normally handle predictable visual context updates, our work offers two unique contributions. First, we provide a general computational framework that can effectively manage a visual context for diverse, unanticipated situations encountered in a user system conversation. Moreover, we optimize the satisfaction of both semantic and visual constraints, which otherwise are difficult to balance using simple heuristics. Second, we present an extensible representation model that uses feature based metrics to uniformly define all constraints. We have applied our work to two different applications and our evaluation has shown the promise of this work.

[1]  I. Biederman Perceiving Real-World Scenes , 1972, Science.

[2]  Yuriko Takeshima,et al.  GADGET/IV: a taxonomic approach to semi-automatic design of information visualization applications using modular visualization environment , 2000, IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings.

[3]  Steven K. Feiner,et al.  View management for virtual and augmented reality , 2001, UIST '01.

[4]  David D. Woods,et al.  Visual Momentum: A Concept to Improve the Cognitive Coupling of Person and Computer , 1984, Int. J. Man Mach. Stud..

[5]  Panos M. Pardalos,et al.  Quadratic Assignment and Related Problems , 1994 .

[6]  Shimei Pan,et al.  Context-based multimodal input understanding in conversational systems , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[7]  Bay-Wei Chang,et al.  Animation: from cartoons to the user interface , 1993, UIST '93.

[8]  Tamara Munzner,et al.  H3: laying out large directed graphs in 3D hyperbolic space , 1997, Proceedings of VIZ '97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium.

[9]  Min Chen,et al.  Automated Generation of Graphic Sketches by Example , 2003, IJCAI.

[10]  J. Stasko,et al.  Focus+context display and navigation techniques for enhancing radial, space-filling hierarchy visualizations , 2000, IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings.

[11]  M. Chun,et al.  Contextual Cueing: Implicit Learning and Memory of Visual Context Guides Spatial Attention , 1998, Cognitive Psychology.

[12]  Patrick Baudisch,et al.  Focus plus context screens: combining display technology with visualization techniques , 2001, UIST '01.

[13]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[14]  Michael L. Mack,et al.  Identifying the Perceptual Dimensions of Visual Complexity of Scenes , 2004 .

[15]  Michelle X. Zhou Visual Planning: A Practical Approach to Automated Presentation Design , 1999, IJCAI.

[16]  Michelle X. Zhou,et al.  An optimization-based approach to dynamic data content selection in intelligent multimedia interfaces , 2004, UIST '04.

[17]  Benjamin B. Bederson,et al.  AppLens and launchTile: two designs for one-handed thumb use on small devices , 2005, CHI.

[18]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[19]  Ben Shneiderman,et al.  Readings in information visualization - using vision to think , 1999 .

[20]  Kevin Mullet,et al.  Designing Visual Interfaces , 1996, SGCH.

[21]  Ben Shneiderman,et al.  Cognitive Layouts of Windows and Multiple Screens for User Interfaces , 1986, Int. J. Man Mach. Stud..