Real-Time Adjustment of Tracking Offsets Through a Brain-Computer Interface for Weight Perception in Virtual Reality

Provision of the perception of pseudo-weight through tracking offsets in virtual reality (VR) allows users to estimate the weight of virtual objects. However, the contribution of the user's real-time perception to such illusions is unknown. Here, we focus on this issue using a brain-computer interface (BCI), through which the user's perception of the weight of virtual objects can be detected in real-time and used to adjust the tracking offset in a closed loop. We first trained a computational model with electroencephalography (EEG) data by asking users to imagine lifting a heavy or a light ball. With this model, the user's perception of the object weight could be detected through the BCI in real-time to adjust the tracking offset, thereby enabling further generation of a more realistic visual sensation. Then, we evaluated the effects of the BCI tracking offset on the perception of the weights of three virtual objects used to simulate real objects, namely, tennis, billiard, and bowling balls. Our results showed that the BCI tracking offset could assist participants in generating perceived weights for virtual objects in VR. We further showed that our approach can provide weight perception through real-time adjustment of the tracking offset, which might be useful for new virtual objects that appear suddenly in the virtual environment. Additionally, most participants (78%) preferred this BCI tracking offset system for weight perception. This article provides the first quantification of weight perception for a virtual object with a BCI that can be used to adjust the tracking offset in real-time for pseudo-weight perception in VR.

[1]  Ping-Keng Jao,et al.  EEG-Based Online Regulation of Difficulty in Simulated Flying , 2023, IEEE Transactions on Affective Computing.

[2]  Pattie Maes,et al.  AttentivU: An EEG-Based Closed-Loop Biofeedback System for Real-Time Monitoring and Improvement of Engagement for Personalized Learning , 2019, Sensors.

[3]  Jinyi Long,et al.  Single-trial prediction of unilateral isometric force levels from EEG signals , 2018, 2018 3rd International Conference on Advanced Robotics and Mechatronics (ICARM).

[4]  Enrico Rukzio,et al.  Breaking the Tracking: Enabling Weight Perception using Perceivable Tracking Offsets , 2018, CHI.

[5]  Yunfa Fu,et al.  Imagined Hand Clenching Force and Speed Modulate Brain Activity and Are Classified by NIRS Combined With EEG , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[6]  Wolfgang Rosenstiel,et al.  Online EEG-Based Workload Adaptation of an Arithmetic Learning Environment , 2017, Front. Hum. Neurosci..

[7]  Antonio Krüger,et al.  Shifty: A Weight-Shifting Dynamic Passive Haptic Proxy to Enhance Object Perception in Virtual Reality , 2017, IEEE Transactions on Visualization and Computer Graphics.

[8]  Jack M. Loomis,et al.  Presence in Virtual Reality and Everyday Life: Immersion within a World of Representation , 2016, PRESENCE: Teleoperators and Virtual Environments.

[9]  Pattie Maes,et al.  GyroVR: Simulating Inertia in Virtual Reality using Head Worn Flywheels , 2016, UIST.

[10]  Sean Follmer,et al.  Wolverine: A Wearable Haptic Interface for Grasping in VR , 2016, UIST.

[11]  Stephen H. Fairclough,et al.  Evaluation of an Adaptive Game that Uses EEG Measures Validated during the Design Process as Inputs to a Biocybernetic Loop , 2016, Front. Hum. Neurosci..

[12]  Daniel Afergan,et al.  Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty Based on Brain State , 2016, CHI.

[13]  Monica A. Perez,et al.  Interhemispheric connectivity during bimanual isometric force generation , 2015, Journal of neurophysiology.

[14]  Makoto Sato,et al.  Wearable 6-DoF wrist haptic device "SPIDAR-W" , 2015, SIGGRAPH Asia Haptic Media And Contents Design.

[15]  Maud Marchal,et al.  Elastic-Arm: Human-scale passive haptic feedback for augmenting interaction and perception in virtual environments , 2015, 2015 IEEE Virtual Reality (VR).

[16]  Manfred Tscheligi,et al.  TorqueScreen: Actuated Flywheels for Ungrounded Kinaesthetic Feedback in Handheld Devices , 2015, TEI.

[17]  Maud Marchal,et al.  Toward "Pseudo-Haptic Avatars": Modifying the Visual Animation of Self-Avatar Can Simulate the Perception of Weight Lifting , 2014, IEEE Transactions on Visualization and Computer Graphics.

[18]  Ipke Wachsmuth,et al.  The Visual, the Auditory and the Haptic - A User Study on Combining Modalities in Virtual Worlds , 2013, HCI.

[19]  Yuanqing Li,et al.  A Hybrid Brain Computer Interface to Control the Direction and Speed of a Simulated or Real Wheelchair , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[20]  Desney S. Tan,et al.  GyroTab: a handheld device that provides reactive torque feedback , 2012, CHI.

[21]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[22]  Gert Pfurtscheller,et al.  Human–Computer Interface Issues in Controlling Virtual Reality With Brain–Computer Interface , 2010, Hum. Comput. Interact..

[23]  J. Blake,et al.  Haptic Glove With MR Brakes for Virtual Reality , 2009, IEEE/ASME Transactions on Mechatronics.

[24]  Kyle N. Winfree,et al.  A high fidelity ungrounded torque feedback device: The iTorqU 2.0 , 2009, World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[25]  Anatole Lécuyer,et al.  Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback , 2009, PRESENCE: Teleoperators and Virtual Environments.

[26]  Olivier Martin,et al.  HEMP-Hand-Displacement-Based Pseudo-Haptics: A Study of a Force Field Application , 2008, 2008 IEEE Symposium on 3D User Interfaces.

[27]  Simon Richir,et al.  Influence of control/display ratio on the perception of mass of manipulated objects in virtual environments , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[28]  Laurent Étienne,et al.  Feeling bumps and holes without a haptic interface: the perception of pseudo-haptic textures , 2004, CHI.

[29]  Hiroo Iwata,et al.  Development of a non-grounded haptic interface using the gyro effect , 2003, 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings..

[30]  Grigore C. Burdea,et al.  The Rutgers Master II-new design force-feedback glove , 2002 .

[31]  Henry Been-Lirn Duh,et al.  Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment , 2002, Proceedings IEEE Virtual Reality 2002.

[32]  Michitaka Hirose,et al.  HapticGEAR: the development of a wearable force display system for immersive projection displays , 2001, Proceedings IEEE Virtual Reality 2001.

[33]  Neff Walker,et al.  Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[34]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.