Multimodality with Eye tracking and Haptics: A New Horizon for Serious Games?

The goal of this review is to illustrate the emerging use of multimodal virtual reality that can benefit learning-based games. The review begins with an introduction to multimodal virtual reality in serious games and we provide a brief discussion of why cognitive processes involved in learning and training are enhanced under immersive virtual environments. We initially outline studies that have used eye tracking and haptic feedback independently in serious games, and then review some innovative applications that have already combined eye tracking and haptic devices in order to provide applicable multimodal frameworks for learning-based games. Finally, some general conclusions are identified and clarified in order to advance current understanding in multimodal serious game production as well as exploring possible areas for new applications.

[1]  Motoyuki Akamatsu,et al.  Movement characteristics using a mouse with tactile and force feedback , 1996, Int. J. Hum. Comput. Stud..

[2]  K. Sunnerhagen,et al.  Virtual reality and haptics as a training device for movement rehabilitation after stroke: a single-case study. , 2004, Archives of Physical Medicine and Rehabilitation.

[3]  Karin Coninx,et al.  The Social Maze: A Collaborative Game to Motivate MS Patients for Upper Limb Training , 2012, ICEC.

[4]  Daniel Kersten,et al.  Bayesian models of object perception , 2003, Current Opinion in Neurobiology.

[5]  Robert J. K. Jacob,et al.  What you look at is what you get: eye movement-based interaction techniques , 1990, CHI '90.

[6]  Laehyun Kim,et al.  Haptic Mouse Interface Actuated by an Electromagnet , 2011, 2011 International Conference on Complex, Intelligent, and Software Intensive Systems.

[7]  Panos Markopoulos,et al.  Motivating arm-hand use for stroke patients by serious games. , 2012, Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference.

[8]  Karin Coninx,et al.  Are existing metaphors in virtual environments suitable for haptic interaction , 2005 .

[9]  Ming C. Lin,et al.  User-centric viewpoint computation for haptic exploration and manipulation , 2001, Proceedings Visualization, 2001. VIS '01..

[10]  Aike Guo,et al.  Crossmodal Interactions Between Olfactory and Visual Learning in Drosophila , 2005, Science.

[11]  E. Granholm,et al.  Pupillometric measures of cognitive and emotional processes. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[12]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[13]  Paul Richard,et al.  Multimodal virtual reality: Input-output devices, system integration, and human factors , 1996, Int. J. Hum. Comput. Interact..

[14]  Allison M. Okamura,et al.  A Magnetically-Actuated Friction Feedback Mouse , 2004 .

[15]  Jochen Triesch,et al.  Saccade contingent updating in virtual reality , 2002, ETRA.

[16]  Mary Jo Dondlinger,et al.  Educational Video Game Design: A Review of the Literature , 2007 .

[17]  Mohan M. Trivedi,et al.  On the Roles of Eye Gaze and Head Dynamics in Predicting Driver's Intent to Change Lanes , 2009, IEEE Transactions on Intelligent Transportation Systems.

[18]  Shumin Zhai,et al.  Manual and gaze input cascaded (MAGIC) pointing , 1999, CHI '99.

[19]  Erik Reinhard,et al.  A psychophysical study of fixation behavior in a computer game , 2008, APGV '08.

[20]  J. Peter Kincaid,et al.  Simulation in education and training , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[21]  Vincent Hayward,et al.  Haptic interfaces and devices , 2004 .

[22]  Bei Yuan,et al.  Blind hero: enabling guitar hero for the visually impaired , 2008, Assets '08.

[23]  M. Bergamasco,et al.  A New Gaze-BCI-Driven Control of an Upper Limb Exoskeleton for Rehabilitation in Real-World Tasks , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[24]  Ming C. Lin,et al.  Introduction to haptic rendering , 2005, SIGGRAPH Courses.

[25]  Gang Luo,et al.  Trajectory prediction of saccadic eye movements using a compressed exponential model. , 2013, Journal of vision.

[26]  Roope Raisamo,et al.  Gaze gestures and haptic feedback in mobile devices , 2014, CHI.

[27]  Abderrahmane Kheddar,et al.  Adaptive Haptic Feedback Steering Wheel for Driving Simulators , 2009, IEEE Transactions on Vehicular Technology.

[28]  Robert S. Allison,et al.  Using Saccadic Suppression to Hide Graphic Updates , 2004, EGVE.

[29]  Ming C. Lin,et al.  Representations and Algorithms for Force-Feedback Display , 2013, Proceedings of the IEEE.

[30]  Kouta Minamizawa,et al.  Gravity grabber: wearable haptic display to present virtual mass sensation , 2007, SIGGRAPH '07.

[31]  Abdulmotaleb El Saddik,et al.  The Role of Haptics in Games , 2012 .

[32]  L. Sentis,et al.  The CHAI Libraries , 2003 .

[33]  Anders Ynnerman,et al.  Designing and Evaluating a Haptic System for Biomolecular Education , 2007, 2007 IEEE Virtual Reality Conference.

[34]  Rainer Groh,et al.  Saccade Detection and Processing for Enhancing 3D Visualizations in Real-Time , 2014, HCI.

[35]  Andrew T. Duchowski,et al.  Gaze-Contingent Displays: A Review , 2004, Cyberpsychology Behav. Soc. Netw..

[36]  Bei Yuan,et al.  Game accessibility: a survey , 2011, Universal Access in the Information Society.

[37]  R.V. Dubey,et al.  Eye-hand coordination assessment/therapy using a robotic haptic device , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[38]  Björn N. S. Vlaskamp,et al.  Crowding degrades saccadic search performance , 2005, Vision Research.

[39]  Andrew T. Duchowski,et al.  Hybrid image-/model-based gaze-contingent rendering , 2007, TAP.

[40]  Lucio Tommaso De Paolis,et al.  Serious Game for Laparoscopic Suturing Training , 2012, CISIS.

[41]  Guang-Zhong Yang,et al.  Enhanced frontoparietal network architectures following “gaze-contingent” versus “free-hand” motor learning , 2013, NeuroImage.

[42]  John M. Henderson,et al.  Predicting Cognitive State from Eye Movements , 2013, PloS one.

[43]  Carrick C. Williams,et al.  Incidental visual memory for targets and distractors in visual search , 2005, Perception & psychophysics.

[44]  Joel C. Perry,et al.  Consistent Arm Rehabilitation from Clinical to Home Environment - Integrating the Universal Haptic Drive into the TeleReha Software Platform , 2013 .

[45]  Guang-Zhong Yang,et al.  Gaze-Contingent Motor Channelling, haptic constraints and associated cognitive demand for robotic MIS , 2012, Medical Image Anal..

[46]  Wen-Bing Horng,et al.  Driver fatigue detection based on eye tracking and dynamk, template matching , 2004, IEEE International Conference on Networking, Sensing and Control, 2004.

[47]  Florian Alt,et al.  Using eye-tracking to support interaction with layered 3D interfaces on stereoscopic displays , 2014, IUI.

[48]  Neff Walker,et al.  Evaluation of the CyberGlove as a whole-hand input device , 1995, TCHI.

[49]  Allison M. Okamura,et al.  Feeling is Believing: Using a Force‐Feedback Joystick to Teach Dynamic Systems , 2000 .

[50]  Toshiyuki Murakami,et al.  Gaze detection based driver modelization in an electric vehicle using virtual force field and Steer by Wire system , 2014, 2014 IEEE 13th International Workshop on Advanced Motion Control (AMC).

[51]  Nigel W. John,et al.  The Role of Haptics in Medical Training Simulators: A Survey of the State of the Art , 2011, IEEE Transactions on Haptics.

[52]  Alex Pentland,et al.  A wearable haptic navigation guidance system , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[53]  Constantine Stephanidis,et al.  Universal access in the information society , 1999, HCI.

[54]  T. C. Nicholas Graham,et al.  Use of eye movements for video game control , 2006, ACE '06.

[55]  Oleg V. Komogortsev,et al.  Can we beat the mouse with MAGIC? , 2013, CHI.

[56]  Domenico Prattichizzo,et al.  The Haptik Library A Component Based Architecture for Uniform Access to Haptic Devices , 2007 .

[57]  Andreas Paepcke,et al.  EyePoint: practical pointing and selection using gaze and keyboard , 2007, CHI.

[58]  John Dingliana,et al.  Collisions and perception , 2001, TOGS.

[59]  Aleš Křenek,et al.  Haptic visualization of molecular model , 2007 .

[60]  R. Mayer,et al.  Interactive Multimodal Learning Environments , 2007 .

[61]  John B. Black,et al.  Learning Newtonian mechanics with an animation game: The role of presentation format on mental model acquisition , 2006 .

[62]  Colin F. Camerer,et al.  Studying learning in games using eye-tracking , 2009 .

[63]  Andrew L. Kun,et al.  Estimating cognitive load using remote eye tracking in a driving simulator , 2010, ETRA.

[64]  John Paulin Hansen,et al.  Evaluation of the Potential of Gaze Input for Game Interaction , 2009, PsychNology J..

[65]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[66]  Daniel R Saunders,et al.  Direct measurement of the system latency of gaze-contingent displays , 2013, Behavior Research Methods.

[67]  Roope Raisamo,et al.  Glasses with haptic feedback of gaze gestures , 2014, CHI Extended Abstracts.

[68]  Sungjae Hwang,et al.  The Haptic steering Wheel: Vibro-tactile based navigation for the driving environment , 2010, 2010 8th IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[69]  Kup-Sze Choi,et al.  Learning Blood Management in Orthopedic Surgery through Gameplay , 2010, IEEE Computer Graphics and Applications.

[70]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[71]  M. Coltheart,et al.  The quarterly journal of experimental psychology , 1985 .

[72]  Tamás D. Gedeon,et al.  "Moving to the centre": A gaze-driven remote camera control for teleoperation , 2011, Interact. Comput..

[73]  Raimund Dachselt,et al.  Still looking: investigating seamless gaze-supported selection, positioning, and manipulation of distant targets , 2013, CHI.

[74]  Louis B. Rosenberg,et al.  Virtual fixtures: Perceptual tools for telerobotic manipulation , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[75]  Gregory J. Zelinsky,et al.  CHAPTER 65 – Specifying the Components of Attention in a Visual Search Task , 2005 .

[76]  Lester C. Loschky,et al.  Gaze-Contingent Multiresolutional Displays: An Integrative Review , 2003, Hum. Factors.

[77]  Pavel Slavík,et al.  Haptic and Spatial Audio Based Navigation of Visually Impaired Users in Virtual Environment Using Low Cost Devices , 2004, User Interfaces for All.

[78]  Cristina Conati,et al.  Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation , 2007, Knowl. Based Syst..

[79]  Hideyuki Ando,et al.  Gaze-Contingent Visual Presentation Based on Remote Saccade Detection , 2007, PRESENCE: Teleoperators and Virtual Environments.

[80]  Anthony Jameson,et al.  Assessing Cognitive Load in Adaptive Hypermedia Systems: Physiological and Behavioral Methods , 2004, AH.

[81]  Benoît Herman,et al.  Toward remote teleoperation with eye and hand: A first experimental study , 2013 .

[82]  Hans-Werner Gellersen,et al.  Cross-device gaze-supported point-to-point content transfer , 2014, ETRA.

[83]  F. Y. Cheng,et al.  Development of an evaluation function for eye-hand coordination robotic therapy , 2011, 2011 IEEE International Conference on Rehabilitation Robotics.

[84]  Guanyang Liu,et al.  Networked multiplayer cooperative interaction using decoupled motion control method in a shared virtual environment with haptic, visual and movement feedback , 2013, Comput. Animat. Virtual Worlds.

[85]  Masashi Toda,et al.  Enhancing Visual Perception Using Dynamic Updating of Display , 2004, Intuitive Human Interfaces for Organizing and Accessing Intellectual Assets.

[86]  Jorge Dias,et al.  Bayesian real-time perception algorithms on GPU , 2010, Journal of Real-Time Image Processing.

[87]  Staffan Björk,et al.  Game Design in Virtual Reality Systems for Stroke Rehabilitation , 2007, MMVR.

[88]  Raimund Dachselt,et al.  Look & touch: gaze-supported target acquisition , 2012, CHI.

[89]  S. Liversedge,et al.  Saccadic eye movements and cognition , 2000, Trends in Cognitive Sciences.

[90]  Harri Ketamo,et al.  Evaluating the usefulness of Eye Tracking in Game-based Learning , 2014, Int. J. Serious Games.

[91]  GedeonTom,et al.  Moving to the centre , 2011 .

[92]  Sim Heng Ong,et al.  Haptics in computer-mediated simulation: Training in vertebroplasty surgery , 2006 .

[93]  Georgios N. Yannakakis,et al.  Towards Adaptive Virtual Camera Control in Computer Games , 2011, Smart Graphics.

[94]  Michael Zyda,et al.  From visual simulation to virtual reality to games , 2005, Computer.

[95]  Anatole Lécuyer,et al.  Using an Eye-Tracking System to Improve Camera Motions and Depth-of-Field Blur Effects in Virtual Environments , 2008, 2008 IEEE Virtual Reality Conference.

[96]  K. Rayner.  The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search , 2009, Quarterly journal of experimental psychology.

[97]  Hongliu Yu,et al.  Motor rehabilitation training after stroke using haptic handwriting and games , 2010 .

[98]  Marc Erich Latoschik,et al.  Evaluation of Binocular Eye Trackers and Algorithms for 3D Gaze Interaction in Virtual Reality Environments , 2008, J. Virtual Real. Broadcast..

[99]  Stefano Gidaro,et al.  The future of telesurgery: a universal system with haptic sensation. , 2012, Journal of the Turkish German Gynecological Association.

[100]  Pei-Chen Sun,et al.  The design of instructional multimedia in e-Learning: A Media Richness Theory-based approach , 2007, Comput. Educ..

[101]  Johann Schrammel,et al.  3D attention: measurement of visual saliency using eye tracking glasses , 2013, CHI Extended Abstracts.

[102]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[103]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[104]  Hua Wang,et al.  Multimodality and Interactivity: Connecting Properties of Serious Games with Educational Outcomes , 2009, Cyberpsychology Behav. Soc. Netw..

[105]  Hideyuki Ando,et al.  Gaze-contingent visual presentation technique with electro-ocular-graph-based saccade detection , 2012, TAP.