Use of a Common Table Architecture for Creating Hands Free, Eyes Free, Noisy Environment (Flex-Modal, Flex-Input) Interfaces

As we look to the design of nomadic systems, we need to find ways to create interfaces that the user can adapt to meet the constraints of different environments. We need to create interfaces that we can use whether we are walking down the hall driving the car, sitting at our workstation, or sitting in a meeting; that we can use when we're under stress or distracted; and that make it easy for us to locate and use new services. This requires an interface that at one point in the day can be used without using vision (e.g., when driving a car), but at other times may need to be totally visual (e.g., on a noisy prop airplane or in a meeting where you cannot use sound). It should also allow you to use a full keyboard and mouse (e.g., when seated) but not require use of these devices since they may not always be usable (e.g., while driving or while walking). Interestingly, this same interface flexibility (including the ability to change modalities) is what is needed to provide access to next generation systems by people with disabilities. That is, if we create fully nomadic systems we will have solved most of the problems and issues around providing access for people with disabilities, with reading problems and for those who are aging. The task in both of these pursuits is identifying ways to create interfaces that are Flex-Modal (the user can chose the sensory modality for information presentation) and Flex-Input (the user can chose which technique they use to input information to the device). Proposed is a method that uses a common table of semantic items that can be built in advance or on the fly. This can then be used as a basis for flex-modal and flex-input interface options.

[1]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[2]  Daniel C. Robbins,et al.  Three-dimensional widgets , 1992, I3D '92.

[3]  Ronald Azuma,et al.  A motion-stabilized outdoor augmented reality system , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[4]  Rudy Darken,et al.  Map usage in virtual environments: orientation issues , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[5]  Aaron Marcus A comparison of graphical user interfaces (excerpt) , 1995 .

[6]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[7]  Doug A. Bowman,et al.  An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments , 1997, SI3D.

[8]  Steve Benford,et al.  A Spatial Model of Interaction in Large Virtual Environments , 1993, ECSCW.

[9]  Chris Culbert,et al.  An expert system development methodology that supports verification and validation , 1989 .

[10]  Dan R. Olsen,et al.  User Interface Management Systems: Models and Algorithms , 1991 .

[11]  J K JacobRobert,et al.  The use of eye movements in human-computer interaction techniques , 1991 .

[12]  Steven K. Feiner,et al.  Windows on the world: 2D windows for 3D augmented reality , 1993, UIST '93.

[13]  Andy Hopper,et al.  The active badge location system , 1992, TOIS.

[14]  Steven K. Feiner,et al.  Language-level support for exploratory programming of distributed virtual environments , 1996, UIST '96.

[15]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.

[16]  James Church,et al.  Wearable sensor badge and sensor jacket for context awareness , 1999, Digest of Papers. Third International Symposium on Wearable Computers.

[17]  Edward O. Thorp,et al.  The invention of the first wearable computer , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[18]  Mark Billinghurst,et al.  A wearable spatial conferencing space , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[19]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[20]  Paul Milgram,et al.  A Taxonomy of Real and Virtual World Display Integration , 1999 .

[21]  John J. Bertin,et al.  The semiology of graphics , 1983 .

[22]  Steven K. Feiner,et al.  Data characterization for automatically visualizing heterogeneous information , 1996, Proceedings IEEE Symposium on Information Visualization '96.

[23]  David A. Rabenhorst,et al.  An Architecture for Virtual Worlds , 1992, Presence: Teleoperators & Virtual Environments.

[24]  H. Petrie,et al.  MOBIC: Designing a Travel Aid for Blind and Elderly People , 1996, Journal of Navigation.

[25]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[26]  Steven Feiner Wearing It Out : First Steps Toward Mobile Augmented Reality Systems , 1999 .

[27]  Gerald Q. Maguire,et al.  Location Aware Mobile Computing , 1997 .

[28]  Christer Carlsson,et al.  DIVE - A platform for multi-user virtual environments , 1993, Comput. Graph..

[29]  Dorée Duncan Seligmann Interactive intent-based illustration : a visual language for 3D worlds , 1993 .

[30]  Chris Schmandt,et al.  Speaking and listening on the run: design for wearable audio computing , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[31]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[32]  Ronald Azuma,et al.  Making augmented reality work outdoors requires hybrid tracking , 1999 .

[33]  Steve Mann,et al.  Wearable Computing: A First Step Toward Personal Imaging , 1997, Computer.

[34]  David E. Breen,et al.  Real-time vision-based camera tracking for augmented reality applications , 1997, VRST '97.

[35]  James D. Foley,et al.  The human factors of computer graphics interaction techniques , 1984, IEEE Computer Graphics and Applications.

[36]  Steven K. Feiner,et al.  Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system , 1999, Comput. Graph..

[37]  Richard Hull,et al.  Towards situated computing , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[38]  Chris Schmandt,et al.  NewsComm: a hand-held interface for interactive access to structured audio , 1996, CHI.

[39]  Gerald Q. Maguire,et al.  IP-Based Protocols for Mobile Internetworking , 1991, SIGCOMM.

[40]  C. Martin,et al.  A Media Taxonomy , 1995, IEEE Multim..

[41]  Steven K. Feiner,et al.  The Representation and Use of a Visual Lexicon for Automated Graphics Generation , 1997, IJCAI.

[42]  Rich Gossweiler,et al.  Virtual Reality on Five Dollars a Day , 1991 .

[43]  Roberta L. Klatzky,et al.  Navigation System for the Blind: Auditory Display Modes and Guidance , 1998, Presence.

[44]  Andrew S. Forsberg,et al.  Image plane interaction techniques in 3D immersive environments , 1997, SI3D.

[45]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[46]  Samuel P. Harbison,et al.  Modula-3 , 1992 .

[47]  Steven K. Feiner,et al.  Of Vampire mirrors and privacy lamps: privacy management in multi-user augmented environments , 1998, UIST '98.

[48]  Kosuke Sato,et al.  MR Aided Engineering: Inspection Support Systems Integrating Virtual Instruments and Process Control , 1999 .

[49]  Steven K. Feiner,et al.  Situated documentaries: embedding multimedia presentations in the real world , 1999, Digest of Papers. Third International Symposium on Wearable Computers.

[50]  Marc Najork,et al.  Obliq-3D: A High-Level, Fast-Turnaround 3D Animation System , 1995, IEEE Trans. Vis. Comput. Graph..

[51]  M. Green Virtual reality user interface: tools and techniques , 1990 .

[52]  Rudy Darken,et al.  A toolset for navigation in virtual environments , 1993, UIST '93.

[53]  Steven K. Feiner,et al.  Exploratory programming of distributed augmented environments , 1999 .

[54]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[55]  Ephraim P. Glinert,et al.  Multimodal Integration , 1996, IEEE Multim..

[56]  Kevin Sowizral,et al.  The Java 3D API Specification , 1997 .

[57]  Ivan Poupyrev,et al.  The go-go interaction technique: non-linear mapping for direct manipulation in VR , 1996, UIST '96.

[58]  Bradley J. Rhodes,et al.  The wearable remembrance agent: A system for augmented memory , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[59]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[60]  Steven K. Feiner,et al.  Automated generation of intent-based 3D Illustrations , 1991, SIGGRAPH.

[61]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[62]  Frederick P. Brooks,et al.  Moving objects in space: exploiting proprioception in virtual-environment interaction , 1997, SIGGRAPH.

[63]  Ronald Azuma,et al.  Tracking in unprepared environments for augmented reality systems , 1999, Comput. Graph..

[64]  Bruce G. Buchanan,et al.  Heuristic DENDRAL - A program for generating explanatory hypotheses in organic chemistry. , 1968 .

[65]  Alex Pentland,et al.  A Wearable Computer Based American Sign Language Recognizer , 1997, SEMWEB.

[66]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[67]  Steven K. Feiner,et al.  Apex: An Experiment in the Automated Creation of Pictorial Explanations , 1985, IEEE Computer Graphics and Applications.

[68]  Richard B. Langley GLONASS: REVIEW AND UPDATE. , 1997 .

[69]  Shree K. Nayar,et al.  Catadioptric omnidirectional camera , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[70]  Niels Ole Bernsen,et al.  Foundations of Multimodal Representations: A Taxonomy of Representational Modalities , 1994, Interact. Comput..

[71]  F. Raab,et al.  Magnetic Position and Orientation Tracking System , 1979, IEEE Transactions on Aerospace and Electronic Systems.

[72]  Jennifer Healey,et al.  Augmented Reality through Wearable Computing , 1997, Presence: Teleoperators & Virtual Environments.

[73]  Ronald Azuma,et al.  The Challenge of Making Augmented Reality Work Outdoors , 1999 .

[74]  James C. Spohrer,et al.  Information in Places , 1999, IBM Syst. J..

[75]  George G. Robertson,et al.  Toolspaces and glances: storing, accessing, and retrieving objects in 3D desktop applications , 1999, SI3D.

[76]  Steinar Kristoffersen,et al.  Designing Interaction Styles for a Mobile Use Context , 1999, HUC.

[77]  Guy L. Steele,et al.  The Java Language Specification , 1996 .

[78]  Neal Lesh,et al.  Indoor navigation using a diverse set of cheap, wearable sensors , 1999, Digest of Papers. Third International Symposium on Wearable Computers.

[79]  Jong-Nam Kim,et al.  An Outdoor Augmented Reality System for GIS Applications , 1999 .

[80]  Reinhold Behringer,et al.  Registration for outdoor augmented reality applications using computer vision techniques and hybrid sensors , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[81]  Douglas A. Bowman,et al.  Interaction Techniques For Common Tasks In Immersive Virtual Environments - Design, Evaluation, And , 1999 .

[82]  Mark Billinghurst,et al.  An evaluation of wearable information spaces , 1998, Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180).

[83]  Michael Gervautz,et al.  Collaborative gaming in augmented reality , 1998, VRST '98.

[84]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[85]  Johan A. Pouwelse,et al.  A feasible low-power augmented-reality terminal , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[86]  Steven K. Feiner,et al.  Future multimedia user interfaces , 1996, Multimedia Systems.

[87]  Richard Mander,et al.  Working with audio: integrating personal tape recorders and desktop computers , 1992, CHI '92.

[88]  M Green Directions for user interface management systems research , 1987, COMG.

[89]  Randy F. Pausch,et al.  Navigation and locomotion in virtual worlds via flight into hand-held miniatures , 1995, SIGGRAPH.

[90]  Blair MacIntyre,et al.  Repo: Obliq with Replicated Objects , 1997 .

[91]  Jock D. Mackinlay Automatic design of graphical presentations , 1987 .

[92]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[93]  David J. Kasik,et al.  A user interface management system , 1982, SIGGRAPH.

[94]  Morton Leonard Heilig,et al.  EL Cine del Futuro: The Cinema of the Future , 1992, Presence: Teleoperators & Virtual Environments.

[95]  Jade Goldstein-Stewart,et al.  Interactive graphic design using automatic presentation knowledge , 1994, CHI '94.

[96]  Wolfgang Wahlster,et al.  WIP: The Automatic Synthesis of Multimodal Presentations , 1991, AAAI Workshop on Intelligent Multimedia Interfaces.

[97]  Mark Billinghurst,et al.  Spatial Information Displays on a Wearable Computer , 1998, IEEE Computer Graphics and Applications.

[98]  Roberta L. Klatzky,et al.  Personal guidance system for the visually impaired , 1994, ASSETS.

[99]  TechnologySankt,et al.  The Responsive Workbench , 1994 .

[100]  Randy Pausch,et al.  Virtual reality on a WIM: interactive worlds in miniature , 1995, CHI '95.

[101]  Katashi Nagao,et al.  The world through the computer: computer augmented interaction with real world environments , 1995, UIST '95.

[102]  Steven K. Feiner,et al.  A distributed 3D graphics library , 1998, SIGGRAPH.

[103]  Morton L. Heilig,et al.  United States Patent Office: stereoscopic-television apparatus for individual use , 1994, COMG.

[104]  Satoru Kawai,et al.  An enhanced treatment of hidden lines , 1987, TOGS.

[105]  Ron Frederick,et al.  Audio aura: light-weight audio augmented reality , 1997, UIST '97.

[106]  Christian Heath,et al.  Mobility in collaboration , 1998, CSCW '98.

[107]  Charles L. Forgy,et al.  Rete: a fast algorithm for the many pattern/many object pattern match problem , 1991 .

[108]  Jun Rekimoto,et al.  Augment-able reality: situated communication through physical and digital spaces , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[109]  Robert J. K. Jacob,et al.  A software model and specification language for non-WIMP user interfaces , 1999, TCHI.

[110]  Thomas A. Furness The Super Cockpit and its Human Factors Challenges , 1986 .

[111]  Borko Furht,et al.  Multimedia systems: an overview , 1994, IEEE MultiMedia.

[112]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[113]  Jock D. Mackinlay,et al.  A morphological analysis of the design space of input devices , 1991, TOIS.

[114]  Mark Billinghurst,et al.  Wearable computers for three dimensional CSCW , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[115]  J. C. R. Licklider,et al.  Man-Computer Symbiosis , 1960 .

[116]  Jakob Nielsen,et al.  Noncommand user interfaces , 1993, CACM.

[117]  Bruce H. Thomas,et al.  Integrating virtual and augmented realities in an outdoor application , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[118]  Brad A. Myers,et al.  A taxonomy of window manager user interfaces , 1988, IEEE Computer Graphics and Applications.

[119]  William Buxton,et al.  Chunking and Phrasing and the Design of Human-Computer Dialogues (Invited Paper) , 1995, IFIP Congress.

[120]  Steven K. Feiner,et al.  Enveloping users and computers in a collaborative 3D augmented reality , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[121]  Steven F. Roth,et al.  Data characterization for intelligent graphics presentation , 1990, CHI '90.

[122]  Steven K. Feiner,et al.  Hybrid user interfaces: breeding virtually bigger interfaces for physically smaller computers , 1991, UIST '91.

[123]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[124]  Luca Cardelli,et al.  A language with distributed scope , 1995, POPL '95.

[125]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[126]  Bruce H. Thomas,et al.  A wearable computer system with augmented reality to support terrestrial navigation , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[127]  Ken Hinckley,et al.  A survey of design issues in spatial input , 1994, UIST '94.