Predicting Future Position From Natural Walking and Eye Movements with Machine Learning

The prediction of human locomotion behavior is a complex task based on data from the given environment and the user. In this study, we trained multiple machine learning models to investigate if data from contemporary virtual reality hardware enables long- and short-term locomotion predictions. To create our data set, 18 participants walked through a virtual environment with different tasks. The recorded positional, orientation- and eye-tracking data was used to train an LSTM model predicting the future walking target. We distinguished between short-term predictions of 50ms and longterm predictions of 2.5 seconds. Moreover, we evaluated GRUs, sequence-to-sequence prediction, and Bayesian model weights. Our results showed that the best short-term model was the LSTM using positional and orientation data with a mean error of 5.14 mm. The best long-term model was the LSTM using positional, orientation and eye-tracking data with a mean error of 65.73 cm. Gaze data offered the greatest predictive utility for long-term predictions of short distances. Our findings indicate that an LSTM model can be used to predict walking paths in VR. Moreover, our results suggest that eye-tracking data provides an advantage for this task.

[1]  Jiwon Seo,et al.  Prediction of Human Trajectory Following a Haptic Robotic Guide Using Recurrent Neural Networks , 2019, 2019 IEEE World Haptics Conference (WHC).

[2]  Ferran Argelaguet,et al.  Virtual vs. Physical Navigation in VR: Study of Gaze and Body Segments Temporal Reorientation Behaviour , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[3]  Tim Fingscheidt,et al.  Analysis of the Effect of Various Input Representations for LSTM-Based Trajectory Prediction , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[4]  A. Patla,et al.  “Look where you’re going!”: gaze behaviour associated with maintaining and changing the direction of locomotion , 2002, Experimental Brain Research.

[5]  James L. Crowley,et al.  Deep learning investigation for chess player attention prediction using eye-tracking and game data , 2019, ETRA.

[6]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[7]  Yoichi Sato,et al.  Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition , 2018, ECCV.

[8]  Shenghua Gao,et al.  Gaze Prediction in Dynamic 360° Immersive Videos , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[10]  Mary M. Hayhoe,et al.  Gaze and the Control of Foot Placement When Walking in Natural Terrain , 2018, Current Biology.

[11]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[12]  Mary C. Whitton,et al.  Walking > walking-in-place > flying, in virtual environments , 1999, SIGGRAPH.

[13]  Paul Sajda,et al.  Sequence Models in Eye Tracking: Predicting Pupil Diameter During Learning , 2020, ETRA Adjunct.

[14]  Eibe Frank,et al.  Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms , 2004, PAKDD.

[15]  Christoph Hölscher,et al.  Do you have to look where you go? Gaze behaviour during spatial decision making , 2011, CogSci.

[16]  Michael J. Black,et al.  On Human Motion Prediction Using Recurrent Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Gerd Bruder,et al.  Exploiting perceptual limitations and illusions to support walking through virtual environments in confined physical spaces , 2013, Displays.

[18]  Andreas M. Kunz,et al.  Eye tracking for locomotion prediction in redirected walking , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).

[19]  Andreas M. Kunz,et al.  Planning redirection techniques for optimal free walking experience using model predictive control , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[20]  E. Hodgson,et al.  Optimizing Constrained-Environment Redirected Walking Instructions Using Search Techniques , 2013, IEEE Transactions on Visualization and Computer Graphics.

[21]  C. G. Keller,et al.  Will the Pedestrian Cross? A Study on Pedestrian Path Prediction , 2014, IEEE Transactions on Intelligent Transportation Systems.

[22]  Dang-Yang Lee,et al.  Path Prediction Using LSTM Network for Redirected Walking , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[23]  Alexandra Kirsch,et al.  Strategies of locomotor collision avoidance. , 2013, Gait & posture.

[24]  Gerd Bruder,et al.  Estimation of Detection Thresholds for Redirected Walking Techniques , 2010, IEEE Transactions on Visualization and Computer Graphics.

[25]  Esa Rahtu,et al.  Digging Deeper Into Egocentric Gaze Prediction , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[26]  Wei Liu,et al.  Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamic , 2018, IJCAI.

[27]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[28]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[29]  Klaus H. Hinrichs,et al.  Taxonomy and Implementation of Redirection Techniques for Ubiquitous Passive Haptic Feedback , 2008, 2008 International Conference on Cyberworlds.

[30]  Sharif Razzaque,et al.  Redirected Walking , 2001, Eurographics.

[31]  G. Courtine,et al.  Human walking along a curved path. I. Body trajectory, segment orientation and the effect of vision , 2003, The European journal of neuroscience.

[32]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[33]  Francesc Moreno-Noguer,et al.  Context-Aware Human Motion Prediction , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Hema Swetha Koppula,et al.  Anticipating Human Activities Using Object Affordances for Reactive Robotic Response , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[36]  Xiongkuo Min,et al.  The prediction of head and eye movement for 360 degree images , 2018, Signal Process. Image Commun..

[37]  Eike Langbehn,et al.  Evaluation of Locomotion Techniques for Room-Scale VR: Joystick, Teleportation, and Redirected Walking , 2018, VRIC.

[38]  Wojciech Matusik,et al.  Eye Tracking for Everyone , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[40]  Rita Cucchiara,et al.  Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model , 2016, IEEE Transactions on Image Processing.

[41]  Otto Lappi,et al.  Humans Use Predictive Gaze Strategies to Target Waypoints for Steering , 2019, Scientific Reports.

[42]  Mark T. Bolas,et al.  Impossible Spaces: Maximizing Natural Walking in Virtual Environments with Self-Overlapping Architecture , 2012, IEEE Transactions on Visualization and Computer Graphics.

[43]  Julien Cornebise,et al.  Weight Uncertainty in Neural Network , 2015, ICML.

[44]  Lucas Theis,et al.  Faster gaze prediction with dense networks and Fisher pruning , 2018, ArXiv.