Windows on the world: 2D windows for 3D augmented reality

INTRODUCTION We describe the design and implementation of a prototype When we think of the use of head-mounted displays and 3D heads-up window system intended for use in a 3D environinteraction devices to present virtual worlds, it is often in ment. Our system includes a see-through head-mounted terms of environments populated solely by 3D objects. display that runs a full X server whose image is overlaid on There are many situations, however, in which 2D text and the user’s view of the physical world. The user’s head is graphics of the sort supported by current window systems tracked so that the display indexes into a large X bitmap, can be useful components of these environments. This is effectively placing the user inside a display space that is especially true in the case of the many applications that run mapped onto part of a surrounding virtual sphere. By under an industry standard window system such as X [13]. tracking the user’s body, and interpreting head motion relaWhile we might imagine porting or enhancing a significant tive to it, we create a portable information surround that X application to take advantage of the 3D capabilities of a envelopes the user as they move about. virtual world, the effort and cost may not be worth the return, especially if the application is inherently 2D. We support three kinds of windows implemented on top of Therefore, we have been exploring how we can incorporate the X server: windows fixed to the head-mounted display, an existing 2D window system within a 3D virtual world. windows fixed to the information surround, and windows fixed to locations and objects in the 3D world. Objects can We are building an experimental system that supports a full also be tracked, allowing windows to move with them. To X11 server on a see-through head-mounted display. Our demonstrate the utility of this model, we describe a small display overlays a selected portion of the X bitmap on the hypermedia system that allows links to be made between user’s view of the world, creating an X-based augmented windows and windows to be attached to objects. Thus, our reality. Depending on the situation and application, the hypermedia system can forge links between any combinauser may wish to treat a window as a stand-alone entity or tion of physical objects and virtual windows. to take advantage of the potential relationships that can be made between it and the visible physical world. To make this possible, we have developed facilities that allow X