Toward user interface virtualization: legacy applications and innovative interaction systems

Single-user, desktop-based computer applications are pervasive in our daily lives and work. The prospect of using these applications with innovative interaction systems, like multi-touch tabletops, tangible user interfaces, large displays or public/private displays, would enable large scale field studies of these technologies, and has the potential to significantly improve their usefulness and, in turn, their availability. This paper focuses on the architectural requirements, design, and implementation of such a technology. First, we review various software technologies for using a single-user desktop application with a different model of user inputs and graphical output. We then present a generic technique for using any closed-source or open-source application with different input and output devices. In our approach, the application is separated from the user input and graphical output subsystem. The core part of the application runs in a system-specific virtual environment. This virtual environment exposes the same API as the removed standard subsystems. This eliminates the need to rewrite the "legacy" application and provides high performances by using the application native way to communicate with the system.

[1]  Elizabeth D. Mynatt,et al.  Mapping GUIs to auditory interfaces , 1992, UIST '92.

[2]  Olivier Chapuis,et al.  Metisse is not a 3D desktop! , 2005, UIST.

[3]  James Gettys,et al.  The X window system , 1990 .

[4]  Jeffrey Nichols,et al.  Interacting at a Distance Using Semantic Snarfing , 2001, UbiComp.

[5]  Olivier Chapuis,et al.  User interface façades: towards fully adaptable user interfaces , 2006, UIST.

[6]  Maneesh Agrawala,et al.  Non-invasive interactive visualization of dynamic architectural environments , 2003, I3D '03.

[7]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[8]  Andy Hopper,et al.  Virtual Network Computing , 1998, IEEE Internet Comput..

[9]  Meredith Ringel Morris,et al.  DiamondSpin: an extensible toolkit for around-the-table interaction , 2004, CHI.

[10]  Bert Brunekreef The PEACE Project. , 1995 .

[11]  W. Keith Edwards,et al.  Systematic output modification in a 2D user interface toolkit , 1997, UIST '97.

[12]  Galen C. Hunt,et al.  Detours: binary interception of Win32 functions , 1999 .

[13]  James Thatcher Screen reader/2: access to OS/2 and the graphical user interface , 1994, Assets '94.

[14]  Terry Winograd,et al.  Fluid interaction with high-resolution wall-size displays , 2001, UIST '01.

[15]  Olivier Chapuis,et al.  Fitts' Law in the Wild: A Field Study of Aimed Movements , 2007 .

[16]  Michel Beaudouin-Lafon,et al.  Novel interaction techniques for overlapping windows , 2001, UIST '01.

[17]  Mary Czerwinski,et al.  Display space usage and window management operation comparisons between single monitor and multiple monitor users , 2004, AVI.

[18]  Hiroshi Ishii,et al.  Emerging frameworks for tangible user interfaces , 2000, IBM Syst. J..

[19]  Peter Parente,et al.  Linux screen reader:: extensible assistive technology , 2006, Assets '06.

[20]  Daniel S. Myers,et al.  Intercepting Arbitrary Functions on Windows , UNIX , and Macintosh OS X Platforms CS-TR-4585 , UMIACS-TR-2004-28 , 2004 .