Eye Gaze-based Early Intent Prediction Utilizing CNN-LSTM

In assistive technologies designed for patients with extremely limited motor or communication capabilities, it is of significant importance to accurately predict the intention of the user, in a timely manner. This paper presents a new framework for the early prediction of the user’s intent via their eye gaze. The seen objects in the displayed images, and the order of their selection are identified from the spatial and temporal information of the gaze. By employing a combination of convolution neuronal network (CNN) and long short term memory (LSTM), early prediction of the user’s intention is enabled. The proposed framework is tested using experimental data obtained from eight subjects. Results demonstrate an average accuracy of 82.27% across all considered intended tasks for early prediction, confirming the effectiveness of the proposed method.

[1]  Gerhard Tröster,et al.  Eye Movement Analysis for Activity Recognition Using Electrooculography , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Laleh Najafizadeh,et al.  Predicting Intention Through Eye Gaze Patterns , 2018, 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[3]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[5]  Bilge Mutlu,et al.  Using gaze patterns to predict task intent in collaboration , 2015, Front. Psychol..

[6]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Jürgen Schmidhuber,et al.  Learning to forget: continual prediction with LSTM , 1999 .

[8]  Weihua Sheng,et al.  A Hidden Markov Model based driver intention prediction system , 2015, 2015 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER).

[9]  J. Hintze,et al.  Violin plots : A box plot-density trace synergism , 1998 .

[10]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Heidar Ali Talebi,et al.  A novel architecture for cooperative remote rehabilitation system , 2016, Comput. Electr. Eng..