HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces

As technology expands and evolves, one-dimensional, graphical user interface (GUI) design becomes increasingly limiting and simplistic. Designers must meet the challenge of developing new and creative interfaces that adapt to meet human needs and technological trends. HCI Beyond the GUI provides designers with this know how by exploring new ways to reach users that involve all of the human senses. Dr. Kortum gathers contributions from leading human factors designers to present a single reference for professionals, researchers, and students. · Explores the human factors involved in the design and implementation of the nontraditional interfaces, detailing design strategies, testing methodologies, and implementation techniques · Provides an invaluable resource for practitioners who design interfaces for children, gamers and users with accessibility needs · Offers extensive case studies, examples and design guidelines

[1]  F. Davide,et al.  Virtual olfactory interfaces : electronic noses and olfactory displays , 2001 .

[2]  Alan Sledd,et al.  Performance Enhancement of a Haptic Arm Exoskeleton , 2006, 2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[3]  Andy Hunt,et al.  Guest Editors' Introduction: An Introduction to Interactive Sonification , 2005, IEEE Multim..

[4]  Kaoru Kohyama,et al.  Chewing patterns of various texture foods studied by electromyography in young and elderly populations , 2002 .

[5]  Eric Horvitz,et al.  S-SEER: Selective Perception in a Multimodal Office Activity Recognition System , 2004, MLMI.

[6]  Ronald S. Fearing,et al.  Human Psychophysics for Teletaction System Design , 2000 .

[7]  Mel Slater,et al.  A Virtual Presence Counter , 2000, Presence: Teleoperators & Virtual Environments.

[8]  Jacques Wainer,et al.  Symgroup: applying social agents in a group interaction system , 2001, GROUP.

[9]  Virginia Best,et al.  Binaural interference and auditory grouping. , 2007, The Journal of the Acoustical Society of America.

[10]  Guy J. Brown,et al.  Computational Auditory Scene Analysis: Principles, Algorithms, and Applications , 2006 .

[11]  Philip H. Marsden,et al.  Allocation of functions , 2004 .

[12]  Eduardo Miranda.  Computer Sound Synthesis for the Electronic Musician , 1998 .

[13]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[14]  Hiroo Iwata,et al.  Pen-based haptic virtual environment , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[15]  R. Axel,et al.  A novel multigene family may encode odorant receptors: A molecular basis for odor recognition , 1991, Cell.

[16]  Teruaki Azuma,et al.  Patterns Observed in the First Chew of Foods with Various Textures , 2001 .

[17]  Yeh-Sun Hong,et al.  A 7 DOF Wearable Robotic Arm Using Pneumatic Actuators , 2001 .

[18]  Bruce N Walker,et al.  Magnitude estimation of conceptual data dimensions for use in sonification. , 2002, Journal of experimental psychology. Applied.

[19]  N. Durlach,et al.  Manual discrimination of force using active finger motion , 1991, Perception & psychophysics.

[20]  Alastair Barrow,et al.  Psychophysical size discrimination using multi-fingered haptic interfaces , 2004 .

[21]  S. Joy Mountford,et al.  The Art of Human-Computer Interface Design , 1990 .

[22]  Alex Waibel,et al.  A framework and toolkit for the construction of multimodal learning interfaces , 1998 .

[23]  Uwe D. Hanebeck,et al.  Motion Compression for Telepresent Walking in Large Target Environments , 2004, Presence: Teleoperators & Virtual Environments.

[24]  Paul Richard,et al.  Human performance evaluation of two handle haptic devices in a dextrous virtual telemanipulation task , 1999, Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289).

[25]  R. Gregory.  The intelligent eye , 1970 .

[26]  Barbara S. Chaparro,et al.  Entering Text Into Hand-Held Devices: Comparing Two Soft Keyboards , 2000 .

[27]  P R Cohen,et al.  The role of voice input for human-machine communication. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Warren Robinett,et al.  Implementation of flying, scaling and grabbing in virtual worlds , 1992, I3D '92.

[29]  Wright-Patterson Afb,et al.  Judging the Orientation of Sinusoidal and Square-Wave Virtual Gratings Presented via 2-DOF and 3-DOF Haptic Interfaces 1 , 2000 .

[30]  Marcia Kilchenman O'Malley,et al.  Comparison of human haptic size discrimination performance in simulated environments with varying levels of force and stiffness , 2004, 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. Proceedings..

[31]  Colby N. Leider.  Digital Audio Workstation , 2004 .

[32]  Clifford Nass,et al.  Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship , 2005 .

[33]  Sharon L. Oviatt,et al.  Mutual disambiguation of recognition errors in a multimodel architecture , 1999, CHI '99.

[34]  Stephen Barrass,et al.  Auditory information design , 1998 .

[35]  Matthew Purver,et al.  Detecting Action Items in Multi-party Meetings: Annotation and Initial Experiments , 2006, MLMI.

[36]  G. Kidd,et al.  Similarity, uncertainty, and masking in the identification of nonspeech auditory patterns. , 2002, The Journal of the Acoustical Society of America.

[37]  R. Howe,et al.  Identification of the mechanical impedance at the human finger tip. , 1997, Journal of biomechanical engineering.

[38]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th Edition) , 2004 .

[39]  Marcia Kilchenman O'Malley,et al.  Passive and active assistance for human performance of a simulated underactuated dynamic task , 2003, 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings..

[40]  K. Iyer,et al.  Sensory design for virtual environments , 2003, SIGGRAPH '03.

[41]  Alex Pentland,et al.  Tactual displays for wearable computing , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[42]  Xing Xie,et al.  Collapse-to-zoom: viewing web pages on small screen devices by interactively removing irrelevant content , 2004, UIST '04.

[43]  Henry Been-Lirn Duh,et al.  “Conflicting” Motion Cues to the Visual and Vestibular Self- Motion Systems Around 0.06 Hz Evoke Simulator Sickness , 2004, Hum. Factors.

[44]  Bruce N. Walker,et al.  PSYCHOPHYSICAL SCALING OF SONIFICATION MAPPINGS: A COMPARISION OF VISUALLY IMPAIRED AND SIGHTED LISTENERS , 2001 .

[45]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[46]  Judith A. Hall.  Nonverbal Behavior, Status, and Gender: How do We Understand Their Relations? , 2006 .

[47]  Martin J. Russell,et al.  Integrating audio and visual information to provide highly robust speech recognition , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[48]  Berry Eggen,et al.  Identifying the intended addressee in mixed human-human and human-computer interaction from non-verbal features , 2005, ICMI '05.

[49]  Thomas B. Moeslund,et al.  Real-time tracking for virtual environments using scaat kalman filtering and unsynchronised cameras , 2006 .

[50]  Gregory D. Abowd,et al.  Human-Computer Interaction (3rd Edition) , 2003 .

[51]  Wolfgang Broll,et al.  Arthur - an augmented reality collaborative design system , 2004 .

[52]  Christophe Ramstein,et al.  Combining haptic and braille technologies: design issues and pilot study , 1996, Assets '96.

[53]  Richard J. Anderson,et al.  Speech, ink, and slides: the interaction of content channels , 2004, MULTIMEDIA '04.

[54]  Fang Chen,et al.  Examining the redundancy of multimodal input , 2006, OZCHI.

[55]  Sharon L. Oviatt,et al.  Predicting spoken disfluencies during human-computer interaction , 1995, Comput. Speech Lang..

[56]  J. M. Rolfe,et al.  Flight simulation , 1988 .

[57]  N I Durlach,et al.  Multidimensional tactile displays: identification of vibratory intensity, frequency, and contactor area. , 1987, The Journal of the Acoustical Society of America.

[58]  Patrick Baudisch,et al.  Summary thumbnails: readable overviews for small screen web browsers , 2005, CHI.

[59]  Carl Gutwin,et al.  Interacting with Big Interfaces on Small Screens: a Comparison of Fisheye, Zoom, and Panning Techniques , 2004, Graphics Interface.

[60]  J. W. Hill.  Study of modeling and evaluation of remote manipulation tasks with force feedback , 1979 .

[61]  N. V. Thakor,et al.  Response to reflected-force feedback to fingers in teleoperations , 1989 .

[62]  Thomas L. Griffiths,et al.  Unsupervised Topic Modelling for Multi-Party Spoken Discourse , 2006, ACL.

[63]  Wolfgang Wahlster,et al.  Dialogue Systems Go Multimodal: The SmartKom Experience , 2006, SmartKom.

[64]  J. Edward Colgate,et al.  Passivity of a class of sampled-data systems: Application to haptic interfaces , 1997 .

[65]  Sharon L. Oviatt,et al.  Multimodal integration patterns in children , 2002, INTERSPEECH.

[66]  Nikolaos G. Tsagarakis,et al.  An integrated tactile/shear feedback array for stimulation of finger mechanoreceptor , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[67]  Saul Greenberg,et al.  Enabling interaction with single user applications through speech and gestures on a multi-user tabletop , 2006, AVI '06.

[68]  M. Srinivasan,et al.  Tactual discrimination of softness. , 1995, Journal of neurophysiology.

[69]  S. Camille Peres,et al.  Auditory graphs: The effects of redundant dimensions and divided attention , 2005 .

[70]  Chris Baber,et al.  Interactive speech technology: human factors issues in the application of speech input/output to computers , 1993 .

[71]  Patrizia Paggio,et al.  Linguistic Interaction in Staging — a Language Engineering View , 2001 .

[72]  Michael R. Falvo.  The nanomanipulator : a teleoperator for manipulating materials at the nanometer scale , 1995 .

[73]  H R BLACKWELL,et al.  Contrast thresholds of the human eye. , 1946, Journal of the Optical Society of America.

[74]  Yacine Bellik,et al.  Multimodal interfaces: new solutions to the problem of computer accessibilty for the blind , 1994, CHI '94.

[75]  Takamichi Nakamoto,et al.  Improvement of olfactory display using solenoid valves , 2007, 2007 IEEE Virtual Reality Conference.

[76]  John Millar Carroll.  HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science , 2003 .

[77]  Kay M. Stanney,et al.  Configural Scoring of Simulator Sickness, Cybersickness and Space Adaptation Syndrome: Similarities and Differences? , 2001 .

[78]  Ahmed Seffah,et al.  Multiple user interfaces : cross-platform applications and context-aware interfaces , 2005 .

[79]  John Makhoul,et al.  DESIGNING CONVERSATIONAL INTERFACES WITH MULTIMODAL INTERACTION , 1998 .

[80]  B. D. Adelstein,et al.  Perceptual decomposition of virtual haptic surfaces , 1993, Proceedings of 1993 IEEE Research Properties in Virtual Reality Symposium.

[81]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[82]  Antonio Frisoli,et al.  A new force-feedback arm exoskeleton for haptic interaction in virtual environments , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[83]  M. Betke,et al.  The Camera Mouse: visual tracking of body features to provide computer access for people with severe disabilities , 2002, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[84]  Lynne E. Bernstein,et al.  For speech perception by humans or machines, three senses are better than one , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[85]  Antti Pirhonen,et al.  NON-SPEECH SOUNDS AS ELEMENTS OF A USE SCENARIO: A SEMIOTIC PERSPECTIVE , 2006 .

[86]  Ali Adjoudani,et al.  Audio-visual speech recognition compared across two architectures , 1995, EUROSPEECH.

[87]  D. Broadbent.  Perception and communication , 1958 .

[88]  Arthur E. Kirkpatrick,et al.  A new approach to haptic augmentation of the GUI , 2006, ICMI '06.

[89]  Vincent Hayward,et al.  Haptic interfaces and devices , 2004 .

[90]  Christopher Frauenberger,et al.  A Generic, Semantically-Based Design Approach for Spatial Auditory Computer Displays , 2004, ICAD.

[91]  Helge Ritter,et al.  BROADCASTING AUDITORY WEATHER REPORTS - A PILOT PROJECT , 2003 .

[92]  K. Sato,et al.  Development of non-constrained master arm with tactile feedback device , 1991, Fifth International Conference on Advanced Robotics 'Robots in Unstructured Environments.

[93]  Thomas A. Furness,et al.  The Effects of the Interface on Navigation in Virtual Environments , 1998 .

[94]  Carolyn Y. Wei,et al.  Towards the perfect infrastructure for usability testing on mobile devices , 2007, CHI Extended Abstracts.

[95]  Woodrow Barfield,et al.  Comments on the Use of Olfactory Displays for Virtual Environments , 1996, Presence: Teleoperators & Virtual Environments.

[96]  L. A. Jones.  Matching Forces: Constant Errors and Differential Thresholds , 1989, Perception.

[97]  George Engelbeck,et al.  Designing Voice Menu Applications for Telephones , 1997 .

[98]  Elizabeth D. Mynatt.  Transforming graphical interfaces into auditory interfaces , 1995, CHI 95 Conference Companion.

[99]  Philip R. Cohen.  The role of natural language in a multimodal interface , 1992, UIST '92.

[100]  Norbert Reithinger,et al.  An Exemplary Interaction with SmartKom , 2006, SmartKom.

[101]  Christian Muller-Tomfelde,et al.  Audio-enhanced collaboration at an interactive electronic whiteboard , 2001 .

[102]  Sharon L. Oviatt,et al.  When do we interact multimodally?: cognitive load and multimodal communication patterns , 2004, ICMI '04.

[103]  A. Macleod,et al.  Quantifying the contribution of vision to speech perception in noise. , 1987, British journal of audiology.

[104]  Paulo Barthelmess,et al.  Edge-splitting in a cumulative multimodal system, for a no-wait temporal threshold on information fusion, combined with an under-specified display , 2006, INTERSPEECH.

[105]  Sharon L. Oviatt,et al.  Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity , 1994, Speech Communication.

[106]  Murray Crease,et al.  GUIDELINES FOR AUDIO-ENHANCEMENT OF GRAPHICAL USER INTERFACE WIDGETS , 2002 .

[107]  Meera Blattner,et al.  Earcons and Icons: Their Structure and Common Design Principles , 1989, Hum. Comput. Interact..

[108]  Sharon L. Oviatt,et al.  Toward a theory of organized multimodal integration patterns during human-computer interaction , 2003, ICMI '03.

[109]  Blake Hannaford,et al.  Manipulation in real, virtual and remote environments , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[110]  Marie-Luce Bourguet,et al.  Towards a taxonomy of error-handling strategies in recognition-based multi-modal human-computer interfaces , 2006, Signal Process..

[111]  Andrea Giachetti,et al.  A holographic collaborative medical visualization system. , 2006, Studies in health technology and informatics.

[112]  Sven Sahle,et al.  SONIFIED EPILEPTIC RHYTHMS , 2006 .

[113]  Sharon L. Oviatt,et al.  Error resolution during multimodal human-computer interaction , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[114]  Hans Jørgen Andersen,et al.  A multispectral approach to robust human skin detection , 2004, CGIV.

[115]  Thomas B. Moeslund,et al.  FGnet Workshop on Visual Observation of Deictic Gestures, Pointing '04, in association with the 17th International Conference on Pattern Recognition, ICPR 2004, 22 August 2004, Cambridge, United Kingdom , 2004 .

[116]  M A Srinivasan,et al.  Manual discrimination of compliance using active pinch grasp: The roles of force and work cues , 1995, Perception & psychophysics.

[117]  Sharon L. Oviatt,et al.  Toward open-microphone engagement for multiparty interactions , 2006, ICMI '06.

[118]  Sharon L. Oviatt,et al.  Multimodal interfaces for dynamic interactive maps , 1996, CHI.

[119]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[120]  Kevin Larson,et al.  Speech Error Correction: The Story of the Alternates List , 2003, Int. J. Speech Technol..

[121]  Anton Nijholt,et al.  Towards the Automatic Generation of Virtual Presenter Agents , 2006, Informing Sci. Int. J. an Emerg. Transdiscipl..

[122]  Sharon L. Oviatt,et al.  Toward Adaptive Information Fusion in Multimodal Systems , 2005, MLMI.

[123]  Jennifer Balogh,et al.  Voice User Interface Design , 2004 .

[124]  Michael M. Cohen,et al.  Modeling Coarticulation in Synthetic Visual Speech , 1993 .

[125]  Sharon L. Oviatt,et al.  Individual differences in multimodal integration patterns: what are they and why do they exist? , 2005, CHI.

[126]  John Kenneth Salisbury,et al.  Making graphics physically tangible , 1999, CACM.

[127]  Nancy J. Cooke,et al.  Varieties of knowledge elicitation techniques , 1994, Int. J. Hum. Comput. Stud..

[128]  Jesse Schell,et al.  Designing Interactive Theme Park Rides , 2001, IEEE Computer Graphics and Applications.

[129]  Matthew Turk,et al.  Perceptual user interfaces (introduction) , 2000, CACM.

[130]  Dennis Wixon,et al.  Book review: PRINCIPLES AND GUIDELINES IN SOFTWARE USER INTERFACE DESIGN by Deborah J . Mayhew (Prentice Hall, 1992) , 1993, SGCH.

[131]  John Bowers,et al.  Assembling the senses: towards the design of cooperative interfaces for visually impaired users , 2004, CSCW.

[132]  Thomas S. Huang,et al.  Scanning the issue - Special issue on human-computer multimodal interface , 2003 .

[133]  John C. Tang.  Findings from Observational Studies of Collaborative Work , 1991, Int. J. Man Mach. Stud..

[134]  K. Toko,et al.  Multichannel taste sensor using electric potential changes in lipid membranes. , 1994, Biosensors & bioelectronics.

[135]  M. Goldfarb,et al.  The effect of virtual surface stiffness on the haptic perception of detail , 2004, IEEE/ASME Transactions on Mechatronics.

[136]  Wei Wang,et al.  A system for analyzing and indexing human-motion databases , 2005, SIGMOD '05.

[137]  Kazuo Tanaka,et al.  Electroencephalogram-based control of an electric wheelchair , 2005, IEEE Transactions on Robotics.

[138]  Lionel Delphin-Poulat,et al.  From a Wizard of Oz experiment to a real time speech and gesture multimodal interface , 2006, Signal Process..

[139]  R. E. Ellis,et al.  Design and evaluation of a high-performance haptic interface , 1996, Robotica.

[140]  Akio Fujimura,et al.  Subclinical alteration of taste sensitivity induced by candesartan in healthy subjects. , 2004, British journal of clinical pharmacology.

[141]  Abhishek Gupta,et al.  Shared Control in Haptic Systems for Performance Enhancement and Training , 2006 .

[142]  Matt Jones,et al.  Mobile Interaction Design , 2006 .

[143]  Dirk Heylen,et al.  Dominance Detection in Meetings Using Easily Obtainable Features , 2005, MLMI.

[144]  J. Vroomen,et al.  Temporal ventriloquism: sound modulates the flash-lag effect. , 2004, Journal of experimental psychology. Human perception and performance.

[145]  John Kenneth Salisbury,et al.  Haptic discrimination of force direction and the influence of visual information , 2006, TAP.

[146]  Mica R. Endsley,et al.  Overcoming Representational Errors in Complex Environments , 2000, Hum. Factors.

[147]  Philip R. Cohen,et al.  Augmenting environments with multimodal interaction , 2003 .

[148]  Septimiu E. Salcudean,et al.  Optimal kinematic design of a haptic pen , 2001 .

[149]  Donald A. Norman,et al.  The invisible computer , 1998 .

[150]  Frederick P. Brooks,et al.  Project GROPEHaptic displays for scientific visualization , 1990, SIGGRAPH.

[151]  Jean-Claude Martin,et al.  Annotating and Measuring Multimodal Behaviour - Tycoon Metrics in the Anvil Tool , 2002, LREC.

[152]  J. Dichgans,et al.  Visual-Vestibular Interaction: Effects on Self-Motion Perception and Postural Control , 1978 .

[153]  R. W. Hukin,et al.  Effectiveness of spatial cues, prosody, and talker characteristics in selective attention. , 2000, The Journal of the Acoustical Society of America.

[154]  A. Patla,et al.  “Look where you’re going!”: gaze behaviour associated with maintaining and changing the direction of locomotion , 2002, Experimental Brain Research.

[155]  F. Barbagli,et al.  Evaluating haptics and 3D stereo displays using Fitts' law , 2004, Proceedings. Second International Conference on Creating, Connecting and Collaborating through Computing.

[156]  Atsuo Takanishi,et al.  Integrated dental robot system for mouth opening and closing training , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[157]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[158]  Perry R. Cook,et al.  Real Sound Synthesis for Interactive Applications , 2002 .

[159]  Michael Goldfarb,et al.  Modeling the human hand as it interacts with a telemanipulation system , 2005 .

[160]  Bernard D. Adelstein,et al.  Demand Characteristics of a Questionnaire Used to Assess Motion Sickness in a Virtual Environment , 2006, IEEE Virtual Reality Conference (VR 2006).

[161]  Eric Vatikiotis-Bateson,et al.  The Dynamics of Audiovisual Behavior in Speech , 1996 .

[162]  Nikolaos G. Tsagarakis,et al.  Development and Control of a ‘Soft-Actuated’ Exoskeleton for Use in Physiotherapy and Training , 2003, Auton. Robots.

[163]  Nigel Bevan,et al.  Methods for Measuring Usability , 1997, INTERACT.

[164]  Shumin Zhai,et al.  Keeping an eye for HCI , 1999, XII Brazilian Symposium on Computer Graphics and Image Processing (Cat. No.PR00481).

[165]  Munsang Kim,et al.  A force reflected exoskeleton-type masterarm for human-robot interaction , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[166]  Joseph J. LaViola,et al.  Hands-free multi-scale navigation in virtual environments , 2001, I3D '01.

[167]  Roberta L. Klatzky,et al.  Comparison of 3-D haptic peg-in-hole tasks in real and virtual environments , 2001, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180).

[168]  Marc L. Resnick,et al.  Effects of Organizational Scheme and Labeling on Task Performance in Product-Centered and User-Centered Retail Web Sites , 2004, Hum. Factors.

[169]  Wolfgang Wahlster,et al.  Plan-based integration of natural language and graphics generation , 1994 .

[170]  Nicholas J. M Patrick.  Design, construction, and testing of a fingertip tactile display for interaction with virtual and remote environments , 1990 .

[171]  Adam Cheyer,et al.  MVIEWS: multimodal tools for the video analyst , 1998, IUI '98.

[172]  Grigore C. Burdea,et al.  Force and Touch Feedback for Virtual Reality , 1996 .

[173]  Vincent Hayward,et al.  Tactile Display Device Using Distributed Lateral Skin Stretch , 2000, Dynamic Systems and Control: Volume 2.

[174]  K. B. Shimoga,et al.  Finger Force and Touch Feedback Issues in Dexterous Telemanipulation , 1992, Proceedings. Fourth Annual Conference on Intelligent Robotic Systems for Space Exploration.

[175]  Trevor Darrell,et al.  Contextual recognition of head gestures , 2005, ICMI '05.

[176]  Robert M. Schumacher.  Phone-Based Interfaces: Research and Guidelines , 1992 .

[177]  P. Bach-y-Rita.  Brain mechanisms in sensory substitution , 1972 .

[178]  I. Hunter,et al.  A perceptual analysis of viscosity , 2004, Experimental Brain Research.

[179]  Mel Slater,et al.  Taking steps: the influence of a walking technique on presence in virtual reality , 1995, TCHI.

[180]  Trevor Darrell,et al.  Hidden Conditional Random Fields for Gesture Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[181]  M. A. Srinivassan.  The impact of visual information on the haptic perception of stiffness in virtual environments , 1996 .

[182]  Keith W. Miller,et al.  How good is good enough?: an ethical analysis of software construction and use , 1994, CACM.

[183]  Philip R. Cohen,et al.  Discourse structure and performance efficiency in interactive and non-interactive spoken modalities☆ , 1991 .

[184]  P. Wiseman.  The Virtual Teacher , 1998 .

[185]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[186]  Abigail Sellen,et al.  The myth of the paperless office , 2001 .

[187]  R.J. Stone.  Advanced human-system interfaces for telerobotics using virtual reality and telepresence technologies , 1991, Fifth International Conference on Advanced Robotics 'Robots in Unstructured Environments.

[188]  Chris Chafe,et al.  Playing by feel: incorporating haptic feedback into computer-based musical instruments , 2001 .

[189]  Yasuhito Suenaga,et al.  "Finger-Pointer": Pointing interface by image processing , 1994, Comput. Graph..

[190]  Helen Petrie,et al.  The use of non-speech sounds in non-visual interfaces to the MS-Windows GUI for blind computer users , 1998 .

[191]  Gerd Kortuem,et al.  Concepts and issues in interfaces for multiple users and multiple devices , 2004 .

[192]  Robert O. Ambrose,et al.  Haptic feedback applications for Robonaut , 2003, Ind. Robot.

[193]  David McGookin,et al.  ADVANTAGES AND ISSUES WITH CONCURRENT AUDIO PRESENTATION AS PART OF AN AUDITORY DISPLAY , 2006 .

[194]  Clare-Marie Karat,et al.  The Beauty of Errors: Patterns of Error Correction in Desktop Speech Systems , 1999, INTERACT.

[195]  Greg Turk,et al.  Interactive simulation in a multi-person virtual world , 1992, CHI.

[196]  Manfred Tscheligi,et al.  Depth and breadth away from the desktop: the optimal information hierarchy for mobile use , 2006, Mobile HCI.

[197]  James J. Troy,et al.  Six degree-of-freedom haptic rendering using voxel sampling , 1999, SIGGRAPH.

[198]  Mel Slater,et al.  Presence in immersive virtual environments , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[199]  James A. Landay,et al.  Visual similarity of pen gestures , 2000, CHI.

[200]  F. P. Brooks,et al.  Grasping reality through illusion—interactive graphics serving science , 1988, CHI '88.

[201]  Alex Pentland,et al.  A wearable haptic navigation guidance system , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[202]  Robert S. Allison,et al.  New simple virtual walking method – walking on the spot , 2004 .

[203]  Robert W. Massof,et al.  Auditory assistive devices for the blind , 2003 .

[204]  Charles A. Baker,et al.  VISUAL PRESENTATION OF INFORMATION , 1954 .

[205]  Robert D. Howe,et al.  Tactile Display of Vibratory Information in Teleoperation and Virtual Environments , 1995, Presence: Teleoperators & Virtual Environments.

[206]  David H. Zald,et al.  A computer-controlled olfactometer for fMRI and electrophysiological studies of olfaction , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[207]  Anton Nijholt,et al.  Pro-active meeting assistants: attention please! , 2007, AI & SOCIETY.

[208]  Bob Fields,et al.  Function allocation: a perspective from studies of work practice , 2000, Int. J. Hum. Comput. Stud..

[209]  William Ribarsky,et al.  Simulator sickness and presence in a high field-of-view virtual environment , 2002, CHI Extended Abstracts.

[210]  J. Edward Colgate,et al.  Design of a high performance haptic interface to virtual environments , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[211]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[212]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[213]  Jef Raskin,et al.  The Humane Interface: New Directions for Designing Interactive Systems , 2000 .

[214]  Blade Kotelly.  Art and Business of Speech Recognition: Creating the Noble Voice , 2003 .

[215]  J M Loomis,et al.  An investigation of tactile hyperacuity. , 1979, Sensory processes.

[216]  Keith Gallagher,et al.  LISTENING TO PROGRAM SLICES , 2006 .

[217]  Chris Schmandt,et al.  Phoneshell: the telephone as computer terminal , 1993, MULTIMEDIA '93.

[218]  Peter Robinson,et al.  The use of gestures in multimodal input , 1998, Assets '98.

[219]  H. Rex Hartson,et al.  Developing user interfaces: ensuring usability through product & process , 1993 .

[220]  Makoto Shimojo,et al.  Human shape recognition performance for 3D tactile display , 1999, IEEE Trans. Syst. Man Cybern. Part A.

[221]  Frank A. Geldard,et al.  Sensory saltation : metastability in the perceptual world , 1977 .

[222]  Cynthia D. Waddell,et al.  Web Accessibility: Web Standards and Regulatory Compliance , 2006 .

[223]  Shumin Zhai,et al.  Manual and gaze input cascaded (MAGIC) pointing , 1999, CHI '99.

[224]  Anurag Kumar Gupta,et al.  Integration patterns during multimodal interaction , 2004, INTERSPEECH.

[225]  Fabio Pianesi,et al.  Automatic detection of group functional roles in face to face interactions , 2006, ICMI '06.

[226]  Ronald Baecker,et al.  Bringing icons to life , 1991, CHI.

[227]  Philip R. Cohen,et al.  Synergistic use of direct manipulation and natural language , 1989, CHI '89.

[228]  Paul ten Have,et al.  Doing Conversation Analysis: A Practical Guide , 1999 .

[229]  Roderick Murray-Smith,et al.  Shoogle: excitatory multimodal interaction on mobile devices , 2007, CHI.

[230]  Paul Duchnowski,et al.  Adaptive bimodal sensor fusion for automatic speechreading , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[231]  Sharon Oviatt,et al.  Designing and evaluating conversational interfaces with animated characters , 2001 .

[232]  Roger W. Cholewiak,et al.  Sensory and Physiological Bases of Touch , 1991 .

[233]  John Kenneth Salisbury,et al.  Haptic Rendering: Introductory Concepts , 2004, IEEE Computer Graphics and Applications.

[234]  Grégory Leplâtre,et al.  How to Tackle Auditory Interface Aesthetics: Discussion and Case Study , 2004, ICAD.

[235]  Joseph Kaye,et al.  Making Scents: aromatic output for HCI , 2004, INTR.

[236]  T B Sheridan,et al.  Teleoperator Performance with Varying Force and Visual Feedback , 1994, Human factors.

[237]  D. McNeill.  Hand and Mind: What Gestures Reveal about Thought , 1992 .

[238]  Luigi Anolli,et al.  A new methodological approach to nonverbal behavior analysis in cultural perspective , 2006, Behavior research methods.

[239]  Bruce Tognazzini,et al.  Tog on Interface , 1992 .

[240]  Minetada Osano,et al.  Modeling Cognitive Structure of Emotion for Developing a Pedagogical Agent Based Interactive Game , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[241]  Allan J. Lichtman.  Ecological Regression Analysis in the Los Angeles County Case and Beyond , 1991 .

[242]  Douglas Hayhoe,et al.  Sorting-Based Menu Categories , 1990, Int. J. Man Mach. Stud..

[243]  Margriet Verlinden,et al.  Multimedia with Animated Sign Language for Deaf Learners , 2005 .

[244]  R V Shannon,et al.  Speech Recognition with Primarily Temporal Cues , 1995, Science.

[245]  Haruo Noma,et al.  SpotScents: A Novel Method of Natural Scent Delivery Using Multiple Scent Projectors , 2006, IEEE Virtual Reality Conference (VR 2006).

[246]  Dilek Z. Hakkani-Tür,et al.  The AT&T WATSON speech recognizer , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[247]  David Demirdjian,et al.  Inferring body pose using speech content , 2005, ICMI '05.

[248]  Hong Z. Tan,et al.  Tactual performance with motional stimulation of the index finger , 1994 .

[249]  Thomas B. Moeslund,et al.  The ARTHUR system: an augmented round table , 2004 .

[250]  M. Whitton,et al.  Review of Four Studies on the Use of Physiological Reaction as a Measure of Presence in StressfulVirtual Environments , 2005, Applied psychophysiology and biofeedback.

[251]  J. Lackner,et al.  Induction of illusory self-rotation and nystagmus by a rotating sound-field. , 1977, Aviation, space, and environmental medicine.

[252]  Sanjeev Kumar,et al.  A multimodal learning interface for sketch, speak and point creation of a schedule chart , 2004, ICMI '04.

[253]  R. T. Verrillo,et al.  Vibrotactile sensitivity and the frequency response of the Pacinian corpuscle , 1966 .

[254]  Fabio Pianesi,et al.  Toward supporting group dynamics , 2006, AIAI.

[255]  Sharon L. Oviatt,et al.  Human-centered design meets cognitive load theory: designing interfaces that help people think , 2006, MM '06.

[256]  Juergen Luettin,et al.  Audio-Visual Automatic Speech Recognition: An Overview , 2004 .

[257]  Randolph G. Bias,et al.  Cost-Justifying Usability: An Update for the Internet Age , 2005 .

[258]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[259]  Joseph F. Dumas,et al.  A Practical Guide to Usability Testing , 1993 .

[260]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[261]  P. Denes.  The Speech Chain , 1963 .

[262]  Steven F. Wiker,et al.  Teleoperator comfort and psychometric stability: Criteria for limiting master-controller forces of operation and feedback during telemanipulation , 1989 .

[263]  Philip R. Cohen,et al.  Multimodal redundancy across handwriting and speech during computer mediated human-human interactions , 2007, CHI.

[264]  Mark R. Mine,et al.  Exploiting Proprioception in Virtual-Environment Interaction , 1997 .

[265]  Richard J. Anderson,et al.  A study of digital ink in lecture presentation , 2004, CHI.

[266]  Catherine Pelachaud,et al.  Audio-visual and multimodal speech-based systems , 2000 .

[267]  Sharon Oviatt,et al.  Multimodal Interfaces , 2008, Encyclopedia of Multimedia.

[268]  Alexander H. Waibel,et al.  The connector: facilitating context-aware communication , 2005, ICMI '05.

[269]  Mark S. Sanders,et al.  Human Factors in Engineering and Design , 2016 .

[270]  Daryle Gardner-Bonneau,et al.  Human Factors and Voice Interactive Systems , 1999 .

[271]  J. Cassell.  Computer Vision for Human–Machine Interaction: A Framework for Gesture Generation and Interpretation , 1998 .

[272]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[273]  Sharon L. Oviatt,et al.  Just Do What I Tell You: The Limited Impact of Instructions on Multimodal Integration Patterns , 2005, User Modeling.

[274]  Allison M. Okamura,et al.  Reality-based models for vibration feedback in virtual environments , 2001 .

[275]  R. Plomp.  Rate of Decay of Auditory Sensation , 1964 .

[276]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[277]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[278]  Y. Matsuoka,et al.  Perceptual limits for a robotic rehabilitation environment using visual feedback distortion , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[279]  J. Ballas.  Common factors in the identification of an assortment of brief everyday sounds , 1993 .

[280]  Hong Z. Tan,et al.  Development of a wearable haptic display for situation awareness in altered-gravity environment: some initial findings , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[281]  Michael Bernard,et al.  Examining a metric for predicting the accessibility of information within hypertext structures , 2002 .

[282]  Renate J. Roske-Hofstrand,et al.  Cognitive networks as a guide to menu organization: An application in the automated cockpit , 1986 .

[283]  Lucy Y. Pao,et al.  Rate-hardness: a new performance metric for haptic interfaces , 2000, IEEE Trans. Robotics Autom..

[284]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[285]  Philip R. Cohen,et al.  Creating tangible interfaces by augmenting physical objects with multimodal language , 2001, IUI '01.

[286]  Hilary Buxton,et al.  Developing Context Sensitive HMM Gesture Recognition , 2003, Gesture Workshop.

[287]  Ephraim P. Glinert,et al.  An experiment into the use of auditory cues to reduce visual workload , 1989, CHI '89.

[288]  R. Fitzpatrick,et al.  The vestibular system , 2005, Current Biology.

[289]  Susan Weinschenk,et al.  Designing effective speech interfaces , 2000 .

[290]  R. Klatzky,et al.  Assessing auditory distance perception using perceptually directed action , 1998, Perception & psychophysics.

[291]  Michael Goldfarb,et al.  The effect of force saturation on the haptic perception of detail , 2002 .

[292]  Edward C. Kaiser,et al.  Using redundant speech and handwriting for learning new vocabulary and understanding abbreviations , 2006, ICMI '06.

[293]  Michèle Courant,et al.  Walking-pad: a step-in-place locomotion interface for virtual environments , 2004, ICMI '04.

[294]  Nancy J. Cooke,et al.  Design of Menus , 1997 .

[295]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[296]  Norbert Reithinger,et al.  Bidirectional Use of Knowledge in the Multi-modal NL Access System XTRA , 1989, IJCAI.

[297]  Chong-Won Lee,et al.  Design of a force reflecting master arm and master hand using pneumatic actuators , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[298]  Sharon L. Oviatt,et al.  Audio-visual cues distinguishing self- from system-directed speech in younger and older adults , 2005, ICMI '05.

[299]  Kay M. Stanney,et al.  Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations , 2004, IEEE Computer Graphics and Applications.

[300]  Michael Johnston,et al.  Finite-state multimodal integration and understanding , 2005, Natural Language Engineering.

[301]  J. Edward Colgate,et al.  Increasing the impedance range of a haptic display by adding electrical damping , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[302]  P. David Stotts,et al.  Facetop tablet:: note-taking assistance for deaf persons , 2006, Assets '06.

[303]  Alexander G. Hauptmann,et al.  Speech and gestures for graphic image manipulation , 1989, CHI '89.

[304]  William K. Durfee,et al.  Influence of haptic and visual displays on the estimation of virtual environment stiffness , 1997 .

[305]  Trevor Darrell,et al.  Constraining human body tracking , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[306]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[307]  Anind K. Dey,et al.  Heuristic evaluation of ambient displays , 2003, CHI '03.

[308]  Sharon L. Oviatt,et al.  User-Centered Modeling for Spoken Language and Multimodal Interfaces , 1996, IEEE Multim..

[309]  John M. Hollerbach,et al.  Haptic manipulation of virtual mechanisms from mechanical CAD designs , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[310]  Eugenia M. Kolasinski,et al.  Simulator Sickness in Virtual Environments. , 1995 .

[311]  R C Williges,et al.  Information Retrieval Strategies File-Search Environment , 1984, Human factors.

[312]  Elizabeth Zoltan-Ford,et al.  How to Get People to Say and Type What Computers Can Understand , 1991, Int. J. Man Mach. Stud..

[313]  Hui Tang,et al.  An oral tactile interface for blind navigation , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[314]  K E Barner,et al.  Design of a haptic data visualization system for people with visual impairments. , 1999, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[315]  John S. Pruitt,et al.  The Persona Lifecycle: Keeping People in Mind Throughout Product Design , 2006 .

[316]  N. Ryba,et al.  The receptors and cells for mammalian taste , 2006, Nature.

[317]  J. S. Nairne.  Remembering over the short-term: the case against the standard model. , 2002, Annual review of psychology.

[318]  Suguru Goto,et al.  The Case Study of An Application of The System, "BodySuit" and "RoboticMusic" - Its Introduction and Aesthetics , 2006, NIME.

[319]  Oliviero Stock,et al.  Multimodal intelligent information presentation , 2005 .

[320]  Marilyn McGee-Lennon,et al.  Audio reminders in the home environment , 2007 .

[321]  Sharon L. Oviatt,et al.  Quiet interfaces that help students think , 2006, UIST.

[322]  Edward C. Kaiser,et al.  Multimodal new vocabulary recognition through speech and handwriting in a whiteboard scheduling application , 2005, IUI.

[323]  Sharon L. Oviatt,et al.  Combining User Modeling and Machine Learning to Predict Users' Multimodal Integration Patterns , 2006, MLMI.

[324]  I. Pollack,et al.  The Information of Elementary Multidimensional Auditory Displays , 1954 .

[325]  Gregory Kramer,et al.  Auditory Display: Sonification, Audification, And Auditory Interfaces , 1994 .

[326]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[327]  Blake Hannaford,et al.  Performance evaluation of a six-axis generalized force-reflecting teleoperator , 1991, IEEE Trans. Syst. Man Cybern..

[328]  Judith Becker Bryant,et al.  Toward a social-cognitive psychology of speech technology: affective responses to speech-based e-service , 2005 .

[329]  K O Johnson,et al.  Tactile spatial resolution. I. Two-point discrimination, gap detection, grating resolution, and letter recognition. , 1981, Journal of neurophysiology.

[330]  Dirk Heylen,et al.  First Steps Towards the Automatic Construction of Argument-Diagrams from Real Discussions , 2006, COMMA.

[331]  Georg v. Békésy,et al.  OLFACTORY ANALOGUE TO DIRECTIONAL HEARING. , 1964 .

[332]  T. Nakamoto,et al.  Study of odor blender using solenoid valves controlled by delta–sigma modulation method for odor recorder , 2002 .

[333]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[334]  Jonathan Klein,et al.  This computer responds to user frustration: Theory, design, and results , 2002, Interact. Comput..

[335]  Gerold Baier,et al.  The Sonification of Rhythms in Human Electroencephalogram , 2004, ICAD.

[336]  M.K. O'Malley,et al.  Design of a haptic arm exoskeleton for training and rehabilitation , 2006, IEEE/ASME Transactions on Mechatronics.

[337]  M. Friedrich,et al.  Harnessing brain signals shows promise for helping patients with paralysis. , 2004, JAMA.

[338]  Vincent Hayward,et al.  Design and multi-objective opti-mization of a link a for a haptic interface , 1994 .

[339]  Patricia S. Denbrook,et al.  Virtual Locomotion: Walking in Place through Virtual Environments , 1999, Presence.

[340]  M. A. Recarte,et al.  Mental workload while driving: effects on visual search, discrimination, and decision making. , 2003, Journal of experimental psychology. Applied.

[341]  Betty J. Mohler,et al.  Visual motion influences locomotion in a treadmill virtual environment , 2004, APGV '04.

[342]  John H. Flowers,et al.  SONIFICATION OF DAILY WEATHER RECORDS: ISSUES OF PERCEPTION, ATTENTION AND MEMORY IN DESIGN CHOICES , 2001 .

[343]  Joëlle Coutaz,et al.  A design space for multimodal systems: concurrent processing and data fusion , 1993, INTERCHI.

[344]  Morton Leonard Heilig,et al.  EL Cine del Futuro: The Cinema of the Future , 1992, Presence: Teleoperators & Virtual Environments.

[345]  Laurence R. Harris,et al.  Vestibular cues and virtual environments: choosing the magnitude of the vestibular cue , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[346]  Bruce N. Walker,et al.  Auditory Navigation Performance is Affected by Waypoint Capture Radius , 2004, ICAD.

[347]  Akio Ando,et al.  Synchronization of speech and hand gestures during multimodal human-computer interaction , 1998, CHI Conference Summary.

[348]  Mary C. Whitton,et al.  Passive haptics significantly enhances virtual environments , 2001 .

[349]  Catherine C. Marshall,et al.  Reading-in-the-small: a study of reading on small form factor devices , 2002, JCDL '02.

[350]  Thomas Hermann,et al.  Sonification for Exploratory Data Analysis , 2002 .

[351]  Anne Guillaume,et al.  Judging the urgency of nonvocal auditory warning signals: perceptual and cognitive processes. , 2003, Journal of experimental psychology. Applied.

[352]  Jefferson Y. Han.  Low-cost multi-touch sensing through frustrated total internal reflection , 2005, UIST.

[353]  Randy Pausch,et al.  Virtual reality on a WIM: interactive worlds in miniature , 1995, CHI '95.

[354]  J. Edward Colgate,et al.  Effects of non-uniform environment damping on haptic perception and performance of aimed movements , 1995 .

[355]  Joëlle Coutaz,et al.  A Wizard of Oz platform for the study of multimodal systems , 1993, INTERCHI Adjunct Proceedings.

[356]  Barbara S. Chaparro,et al.  Examining the Effects of Hypertext Shape on User Performance , 2002 .

[357]  Joseph J. LaViola,et al.  A discussion of cybersickness in virtual environments , 2000, SGCH.

[358]  Regine Freitag,et al.  Making Use of Scenarios for Validating Analysis and Design , 1998, IEEE Trans. Software Eng..

[359]  Vincent Hayward,et al.  Performance Measures for Haptic Interfaces , 1996 .

[360]  S. Blumstein,et al.  Phonetic features and acoustic invariance in speech , 1981, Cognition.

[361]  Petr Janata,et al.  Marketbuzz: Sonification of real-time financial dataa , 2004 .

[362]  James N. MacGregor.  A Comparison of the Effects of Icons and Descriptors in a Videotex Menu Retrieval , 1992, Int. J. Man Mach. Stud..

[363]  Sharon L. Oviatt,et al.  Referential features and linguistic indirection in multimodal language , 1998, ICSLP.

[364]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[365]  Marcia Kilchenman O'Malley,et al.  A Study of Perceptual Performance in Haptic Virtual Environments , 2006, J. Robotics Mechatronics.

[366]  Jakob Nielsen,et al.  The usability engineering life cycle , 1992, Computer.

[367]  Li Deng,et al.  Distributed speech processing in miPad's multimodal user interface , 2002, IEEE Trans. Speech Audio Process..

[368]  Amar Ramdane-Cherif,et al.  A ubiquitous context-sensitive multimodal multimedia computing system and its machine learning-based reconfiguration at the architectural level , 2005, Seventh IEEE International Symposium on Multimedia (ISM'05).

[369]  N. Hogan,et al.  Robot-aided neurorehabilitation. , 1998, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[370]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[371]  Yacine Bellik,et al.  A multimodal dialogue controller for multimodal user interface management system application: a multimodal window manager , 1993, CHI '93.

[372]  Robert S. Kennedy,et al.  Implications of balance disturbances following exposure to virtual reality systems , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[373]  Joëlle Coutaz,et al.  Applying the Wizard of Oz Technique to the Study of Multimodal Systems , 1993, EWHCI.

[374]  Usman Haque.  Scents of space: an interactive smell system , 2004, SIGGRAPH '04.

[375]  Helge Ritter,et al.  Vocal sonification of pathologic EEG features , 2006 .

[376]  A. Baddeley.  Working memory: looking back and looking forward , 2003, Nature Reviews Neuroscience.

[377]  Marcia Kilchenman O'Malley,et al.  On the Ability of Humans to Haptically Identify and Discriminate Real and Simulated Objects , 2005, Presence: Teleoperators & Virtual Environments.

[378]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[379]  Hiroo Iwata,et al.  Walking about virtual environments on an infinite floor , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[380]  N. Tsagarakis,et al.  A 7 DOF pneumatic muscle actuator (pMA) powered exoskeleton , 1999, 8th IEEE International Workshop on Robot and Human Interaction. RO-MAN '99 (Cat. No.99TH8483).

[381]  J. Edward Colgate,et al.  Factors affecting the Z-Width of a haptic display , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[382]  Andreas Paepcke,et al.  Seeing the whole in parts: text summarization for web browsing on handheld devices , 2001, WWW '01.

[383]  Cagatay Basdogan,et al.  Surgical Simulation: An Emerging Technology for Training in Emergency Medicine , 1997, Presence: Teleoperators & Virtual Environments.

[384]  S. S. Stevens.  On the psychophysical law. , 1957, Psychological review.

[385]  Philip R. Cohen,et al.  A visual modality for the augmentation of paper , 2001, PUI '01.

[386]  Thomas B. Moeslund,et al.  A Natural Interface to a Virtual Environment through Computer Vision-Estimated Pointing Gestures , 2001, Gesture Workshop.

[387]  Sara Bly,et al.  Presenting information in sound , 1982, CHI '82.

[388]  William H. Warren,et al.  Optic flow is used to control human walking , 2001, Nature Neuroscience.

[389]  Vincent Hayward,et al.  Multiple-goal kinematic optimization of a parallel spherical mechanism with actuator redundancy , 1992, IEEE Trans. Robotics Autom..

[390]  Kwan Min Lee,et al.  Speech Versus Touch: A Comparative Study of the Use of Speech and DTMF Keypad for Navigation , 2005, Int. J. Hum. Comput. Interact..

[391]  Philip R. Cohen,et al.  Unification-based multimodal integration , 1997 .

[392]  Grigore C. Burdea,et al.  The Rutgers Master II-new design force-feedback glove , 2002 .

[393]  Pernilla Ulfvengren,et al.  Design of Natural Warning Sounds in Human-Machine Systems , 2003 .

[394]  Michael W. Haas,et al.  Postural Instability and Motion Sickness in a Fixed-Base Flight Simulator , 2000, Hum. Factors.

[395]  Xiao Huang,et al.  Distributed pointing for multimodal collaboration over sketched diagrams , 2005, ICMI '05.

[396]  P A Kolers,et al.  Readability of Text Scrolled on Visual Display Terminals as a Function of Window Size , 1983, Human factors.

[397]  James A. Ballas,et al.  The Design of Mixed Use Virtual Auditory Displays: Recent Findings with a Dual-Task Paradigm , 2004, ICAD.

[398]  Peter Robinson,et al.  Using haptic feedback to enhance computer interaction for motion-impaired users , 2000 .

[399]  Gordon Rugg,et al.  The sorting techniques: a tutorial paper on card sorts, picture sorts and item sorts , 1997 .

[400]  Dale A. Lawrence,et al.  Performance trade-offs for hand controller design , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[401]  Philip R. Cohen,et al.  Collaborative multimodal photo annotation over digital paper , 2006, ICMI '06.

[402]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[403]  S. Cobb,et al.  Static posture tests for the assessment of postural instability after virtual environment use , 1998, Brain Research Bulletin.

[404]  S. McAdams,et al.  Auditory Cognition. (Book Reviews: Thinking in Sound. The Cognitive Psychology of Human Audition.) , 1993 .

[405]  Hong Z. Tan,et al.  HUMAN FACTORS FOR THE DESIGN OF FORCE-REFLECTING HAPTIC INTERFACES , 1994 .

[406]  Karun B. Shimoga,et al.  A survey of perceptual feedback issues in dexterous telemanipulation. II. Finger touch feedback , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[407]  Ronald Poppe,et al.  Vision-based human motion analysis: An overview , 2007, Comput. Vis. Image Underst..

[408]  Philip R. Cohen,et al.  Tangible multimodal interfaces for safety-critical applications , 2004, CACM.

[409]  H. Moskowitz.  Umami: A basic taste Y. Kamamura and M. R. Kare (Eds.). Pp. 649, New York, Marcel Dekker, Inc., 1986, $65.00 , 1987, Appetite.

[410]  Oren Etzioni,et al.  A reliable natural language interface to household appliances , 2003, IUI '03.

[411]  Hideki Hashimoto,et al.  7 DOF arm type haptic interface for teleoperation and virtual reality systems , 1998, Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No.98CH36190).

[412]  Bruce N. Walker,et al.  SPEARCONS: SPEECH-BASED EARCONS IMPROVE NAVIGATION PERFORMANCE IN AUDITORY MENUS , 2006 .

[413]  Jorge Angeles,et al.  Kinematic Isotropy and the Optimum Design of Parallel Manipulators , 1997, Int. J. Robotics Res..

[414]  David McGookin,et al.  Haptic and Audio Interaction Design: First International Workshop, HAID 2006, Glasgow, UK, August 31-September 1, 2006, Proceedings , 2006 .

[415]  Mark H. Draper,et al.  The adaptive effects of virtual interfaces: vestibulo-ocular reflex and simulator sickness , 1998 .

[416]  Jean-Baptiste de la Rivière,et al.  Hand Postures Recognition in Large - Display VR Environments , 2003, Gesture Workshop.

[417]  Jorge Angeles,et al.  Optimum design of manipulators under dynamic isotropy conditions , 1993, [1993] Proceedings IEEE International Conference on Robotics and Automation.

[418]  I. Hunter,et al.  A perceptual analysis of stiffness , 2004, Experimental Brain Research.

[419]  René Farcy,et al.  Comparison of Various Interface Modalities for a Locomotion Assistance Device , 2002, ICCHP.

[420]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[421]  Ville Pulkki,et al.  Virtual Sound Source Positioning Using Vector Base Amplitude Panning , 1997 .

[422]  Alexander H. Waibel,et al.  Model-based and empirical evaluation of multimodal interactive error correction , 1999, CHI '99.

[423]  Tanja Schultz,et al.  Identifying the addressee in human-human-robot interactions based on head pose and speech , 2004, ICMI '04.

[424]  Sharon Oviatt,et al.  Multimodal interactive maps: designing for human performance , 1997 .

[425]  Misha Pavel,et al.  Context Shifts: Extending the Meanings of Physical Objects With Language , 2001, Hum. Comput. Interact..

[426]  Linda J. Ferrier,et al.  Using the Baby-Babble-Blanket for infants with motor problems: an empirical study , 1994, Assets '94.

[427]  Lynette A. Jones,et al.  Development of a tactile vest , 2004, 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. Proceedings..

[428]  Jerome Pasquero.  Survey on Communication through Touch , 2006 .

[429]  C D Wickens,et al.  Compatibility and Resource Competition between Modalities of Input, Central Processing, and Output , 1983, Human factors.

[430]  Anton Nijholt,et al.  Addressee Identification in Face-to-Face Meetings , 2006, EACL.

[431]  Susan Weinschenk,et al.  GUI Design Essentials , 1997 .

[432]  Yacine Bellik,et al.  Media integration in multimodal interfaces , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[433]  Won S. Kim.  A new scheme of force reflecting control , 1992 .

[434]  Karon E. MacLean,et al.  Perceiving ordinal data haptically under workload , 2005, ICMI '05.

[435]  Nikolaos G. Tsagarakis,et al.  A haptic-enabled multimodal interface for the planning of hip arthroplasty , 2006, IEEE MultiMedia.

[436]  Sharif Razzaque,et al.  Redirected Walking , 2001, Eurographics.

[437]  Louis B. Rosenberg,et al.  Virtual fixtures: Perceptual tools for telerobotic manipulation , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[438]  Thomas B. Moeslund,et al.  Sequential Monte Carlo tracking of body parameters in a sub-space , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[439]  Rudy Darken,et al.  The omni-directional treadmill: a locomotion device for virtual worlds , 1997, UIST '97.

[440]  Tosiyasu L. Kunii,et al.  Model-based analysis of hand posture , 1995, IEEE Computer Graphics and Applications.

[441]  Sadaoki Furui,et al.  An Overview of Speaker Recognition Technology , 1996 .

[442]  Caroline Hummels,et al.  Meaningful gestures for human computer interaction: beyond hand postures , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[443]  Massimo Bergamasco,et al.  An arm exoskeleton system for teleoperation and virtual environments applications , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[444]  Luis Enrique Sucar,et al.  Visual Recognition of Similar Gestures , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[445]  Sharon L. Oviatt,et al.  Integration themes in multimodal human-computer interaction , 1994, ICSLP.

[446]  E. Schegloff,et al.  The preference for self-correction in the organization of repair in conversation , 1977 .

[447]  W. M. Rabinowitz,et al.  Manual discrimination and identification of length by the finger-span method , 1989, Perception & psychophysics.

[448]  Brad A. Myers,et al.  WebThumb: interaction techniques for small-screen browsers , 2002, UIST '02.

[449]  A. Mills.  On the minimum audible angle , 1958 .

[450]  Heinrich H. Bülthoff,et al.  Visual Homing Is Possible Without Landmarks: A Path Integration Study in Virtual Reality , 2002, Presence: Teleoperators & Virtual Environments.

[451]  Thomas B. Moeslund,et al.  Action Recognition Using Motion Primitives and Probabilistic Edit Distance , 2006, AMDO.

[452]  Stephen A. Brewster,et al.  Haptic Graphs for Blind Computer Users , 2000, Haptic Human-Computer Interaction.

[453]  Q. Summerfield,et al.  Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. , 1985, The Journal of the Acoustical Society of America.

[454]  Mandayam A. Srinivasan,et al.  Force Shading for Haptic Shape Perception , 1996, Dynamic Systems and Control.

[455]  Philip R. Cohen,et al.  The Emergence of Representations in Collaborative Space Planning over Digital Paper: Preliminary Observations , 2006 .

[456]  F A Mussa-Ivaldi,et al.  Haptic illusions: experiments on human manipulation and perception of "virtual objects". , 1990, Cold Spring Harbor symposia on quantitative biology.

[457]  J. F. Meech,et al.  User requirements when interacting with virtual objects , 1996 .

[458]  Septimiu E. Salcudean,et al.  A coarse-fine approach to force-reflecting hand controller design , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[459]  Q. Summerfield,et al.  Lipreading and audio-visual speech perception. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[460]  Michael J. Black,et al.  Assistive technology and robotic control using motor cortex ensemble‐based neural interface systems in humans with tetraplegia , 2007, The Journal of physiology.

[461]  Allison M. Okamura,et al.  Vibration feedback models for virtual environments , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[462]  Bruce Balentine,et al.  How to Build a Speech Recognition Application , 1999 .

[463]  A. Descloux,et al.  1982/83 End office connection study: Analog voice and voiceband data transmission performance characterization of the public switched network , 1984, AT&T Bell Laboratories Technical Journal.

[464]  Suprateek Sarker,et al.  Understanding mobile handheld device use and adoption , 2003, CACM.

[465]  Ahmed Seffah,et al.  Multi-devices "Multiple" user interfaces: development models and research opportunities , 2004, J. Syst. Softw..

[466]  Li Deng,et al.  MiPad: a multimodal interaction prototype , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[467]  Vladimir Pavlovic,et al.  Integration of audio/visual information for use in human-computer intelligent interaction , 1997, Proceedings of International Conference on Image Processing.

[468]  Jonathan Grudin,et al.  Why CSCW applications fail: problems in the design and evaluationof organizational interfaces , 1988, CSCW '88.

[469]  Tammo Houtgast,et al.  Auditory distance perception in rooms , 1999, Nature.

[470]  E. Schegloff,et al.  A simplest systematics for the organization of turn-taking for conversation , 1974 .

[471]  Joëlle Coutaz,et al.  A generic platform for addressing the multimodal challenge , 1995, CHI '95.

[472]  Bernhard Suhm,et al.  Multimodal interactive error recovery for non-conversational speech user interfaces , 1999 .

[473]  D. Rempel,et al.  Effects of finger posture on carpal tunnel pressure during wrist motion. , 1998, The Journal of hand surgery.

[474]  Doug A. Bowman,et al.  Travel in immersive virtual environments: an evaluation of viewpoint motion control techniques , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[475]  S. E. Salcudean,et al.  On the Emulation of Stiff Walls and Static Friction with a Magnetically Levitated Input/Output Devic , 1997 .

[476]  A. Szczesniak.  Texture is a sensory property , 2002 .

[477]  Clarence A. Ellis,et al.  The Neem Platform: An Evolvable Framework for Perceptual Collaborative Applications , 2005, Journal of Intelligent Information Systems.

[478]  Juergen Luettin,et al.  Audio-Visual Speech Modeling for Continuous Speech Recognition , 2000, IEEE Trans. Multim..

[479]  Takamichi Nakamoto,et al.  Study of odor recorder for dynamical change of odor using QCM sensors and neural network , 2002 .

[480]  Sharon L. Oviatt,et al.  Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences , 2003, ICMI '03.

[481]  Thomas Rist,et al.  The PPP persona: a multipurpose animated presentation agent , 1996, AVI '96.

[482]  Ernst L. Leiss,et al.  Kinesthetic and visual force display for telerobotics , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[483]  Albert S. Bregman,et al.  The Auditory Scene. (Book Reviews: Auditory Scene Analysis. The Perceptual Organization of Sound.) , 1990 .

[484]  Thomas H. Massie,et al.  The PHANToM Haptic Interface: A Device for Probing Virtual Objects , 1994 .

[485]  J. Amoore.  Molecular basis of odor , 1970 .

[486]  Eric Martinson,et al.  Improving human-robot interaction through adaptation to the auditory scene , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[487]  J. Sweller,et al.  Reducing cognitive load by mixing auditory and visual presentation modes , 1995 .

[488]  R. J. Irwin,et al.  Differential thresholds for limb movement measured using adaptive techniques , 1992, Perception & psychophysics.

[489]  Mary C. Whitton,et al.  Walking > walking-in-place > flying, in virtual environments , 1999, SIGGRAPH.

[490]  Hiroo Iwata,et al.  Haptic interfaces , 2002 .

[491]  Sarah A. Douglas,et al.  Application-based evaluation of haptic interfaces , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[492]  Alexander I. Rudnicky,et al.  Using simple speech-based features to detect the state of a meeting and the roles of the meeting participants , 2004, INTERSPEECH.

[493]  Ho-Sub Yoon,et al.  Gesture-based editing system for graphic primitives and alphanumeric characters , 1999 .

[494]  F. J. Clark,et al.  HOW ACCURATELY CAN WE PERCEIVE THE POSITIONS OF OUR LIMBS , 1992 .

[495]  Kara A. Latorella,et al.  The Scope and Importance of Human Interruption in Human-Computer Interaction Design , 2002, Hum. Comput. Interact..

[496]  A. Szczesniak.  Classification of textural characteristics , 1963 .

[497]  J. V. Draper,et al.  The implications of force reflection for teleoperation in space , 1987 .

[498]  S. Bolanowski,et al.  Four channels mediate the mechanical aspects of touch. , 1988, The Journal of the Acoustical Society of America.

[499]  Marcia Kilchenman O'Malley,et al.  Force saturation, system bandwidth, information transfer, and surface quality in haptic interfaces , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[500]  Tomohiro Tanikawa,et al.  Wearable Olfactory Display: Using Odor in Outdoor Environment , 2006, IEEE Virtual Reality Conference (VR 2006).

[501]  Jakob Nielsen,et al.  Designing Web Usability: The Practice of Simplicity , 1999 .

[502]  Sharon L. Oviatt,et al.  Prototyping novel collaborative multimodal systems: simulation, data collection and analysis tools for the next decade , 2006, ICMI '06.

[503]  Anton Nijholt,et al.  Online and off-line visualization of meeting information and meeting support , 2006, The Visual Computer.

[504]  T. L. Brooks,et al.  Telerobotic response requirements , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[505]  H. Schiffman.  Sensation and Perception: An Integrated Approach , 1976 .

[506]  Jr. J.P. Campbell,et al.  Speaker recognition: a tutorial , 1997, Proc. IEEE.

[507]  Andy Hunt,et al.  The Importance of Interaction in Sonification , 2004, ICAD.

[508]  James N. Templeman,et al.  Pointman - A Device-Based Control for Realistic Tactical Movement , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[509]  Minh Tue Vo,et al.  Building an application framework for speech and pen input integration in multimodal learning interfaces , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[510]  Sharon L. Oviatt,et al.  Breaking the Robustness Barrier: Recent Progress on the Design of Robust Multimodal Systems , 2002, Adv. Comput..

[511]  Robert D. Howe,et al.  A tactile sensor for localizing transient events in manipulation , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[512]  Sharon L. Oviatt,et al.  A rapid semi-automatic simulation technique for investigating interactive speech and handwriting , 1992, ICSLP.

[513]  Dirk Heylen,et al.  DIALOGUE-ACT TAGGING USING SMART FEATURE SELECTION; RESULTS ON MULTIPLE CORPORA , 2006, 2006 IEEE Spoken Language Technology Workshop.

[514]  Simon Carlile,et al.  THE PHYSICAL AND PSYCHOPHYSICAL BASIS OF SOUND LOCALIZATION , 1996 .

[515]  Günther Schmidt,et al.  Application-specific evaluation of tactile array displays for the human fingertip , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[516]  Alexandrina Rogozan,et al.  Adaptive fusion of acoustic and visual sources for automatic speech recognition , 1998, Speech Commun..

[517]  Sharon L. Oviatt,et al.  Taming recognition errors with a multimodal interface , 2000, CACM.

[518]  Ehsan Samei,et al.  Liquid-crystal displays for medical imaging: a discussion of monochrome versus color , 2004, Medical Imaging: Image-Guided Procedures.

[519]  J Robert-Ribes,et al.  Complementarity and synergy in bimodal speech: auditory, visual, and audio-visual identification of French oral vowels in noise. , 1998, The Journal of the Acoustical Society of America.

[520]  Hiroo Iwata,et al.  Path Reproduction Tests Using a Torus Treadmill , 1999, Presence.

[521]  F. Heider.  The psychology of interpersonal relations , 1958 .

[522]  Takamichi Nakamoto,et al.  Active gas/odor sensing system using automatically controlled gas blender and numerical optimization technique , 1994 .