Confusion Prediction from Eye-Tracking Data: Experiments with Machine Learning

Predicting user confusion can help improve information presentation on websites, mobile apps, and virtual reality interfaces. One promising information source for such prediction is eye-tracking data about gaze movements on the screen. Coupled with think-aloud records, we explore if user's confusion is correlated with primarily fixation-level features. We find that random forest achieves an accuracy of more than 70% when prediction user confusion using only fixation features. In addition, adding user-level features (age and gender) improves the accuracy to more than 90%. We also find that balancing the classes before training improves performance. We test two balancing algorithms, Synthetic Minority Over Sampling Technique (SMOTE) and Adaptive Synthetic Sampling (ADASYN) finding that SMOTE provides a higher performance increase. Overall, this research contains implications for researchers interested in inferring users' cognitive states from eye-tracking data.

[1]  Bernard J. Jansen,et al.  “Is More Better?”: Impact of Multiple Photos on Perception of Persona Profiles , 2018, CHI.

[2]  Bernard J. Jansen,et al.  Findings of a User Study of Automatically Generated Personas , 2018, CHI Extended Abstracts.

[3]  Yeliz Yesilada,et al.  Eye tracking scanpath analysis techniques on web pages: A survey, evaluation and comparison , 2015 .

[4]  R. Fisher.  Social Desirability Bias and the Validity of Indirect Questioning , 1993 .

[5]  Thorsten Joachims,et al.  The influence of task and gender on search and evaluation behavior using Google , 2006, Inf. Process. Manag..

[6]  Qian Guo,et al.  A diary study of information encountering triggered by visual stimuli on micro-blogging services , 2019, Inf. Process. Manag..

[7]  Hahn-Ming Lee,et al.  Automated estimation of item difficulty for multiple-choice tests: An application of word embedding techniques , 2018, Inf. Process. Manag..

[8]  Kellie J. Archer,et al.  Empirical characterization of random forest variable importance measures , 2008, Comput. Stat. Data Anal..

[9]  Bernard J. Jansen,et al.  Confusion and information triggered by photos in persona profiles , 2019, Int. J. Hum. Comput. Stud..

[10]  Bernard J. Jansen,et al.  Design Issues in Automatically Generated Persona Profiles: A Qualitative Analysis from 38 Think-Aloud Transcripts , 2019, CHIIR.

[11]  Bernard J. Jansen,et al.  Validating social media data for automatic persona generation , 2016, 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA).

[12]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[13]  Bernard J. Jansen,et al.  Imaginary People Representing Real Numbers , 2018, ACM Trans. Web.

[14]  S. M. Brown,et al.  Note on the Cochran Q Test , 1970 .

[15]  Bernard J. Jansen,et al.  Automatic Persona Generation for Online Content Creators: Conceptual Rationale and a Research Agenda , 2019, Personas - User Focused Design.

[16]  Pam J. Mayhew,et al.  To Intervene or Not to Intervene:An Investigation of Three Think-Aloud Protocols in Usability Testing , 2017 .

[17]  Stephanie Wilson,et al.  Identifying web usability problems from eye-tracking data , 2007, BCS HCI.

[18]  O. Gambini,et al.  An artificial neural network that uses eye-tracking performance to identify patients with schizophrenia. , 1999, Schizophrenia bulletin.

[19]  Michelle A. Borkin,et al.  Eye Fixation Metrics for Large Scale Evaluation and Comparison of Information Visualizations , 2015, ETVIS.

[20]  FerhatOnur,et al.  Low Cost Eye Tracking , 2016 .

[21]  Eun Yi Kim,et al.  Eye Tracking Using Neural Network and Mean-Shift , 2006, ICCSA.

[22]  Bernard J. Jansen,et al.  Persona Generation from Aggregated Social Media Data , 2017, CHI Extended Abstracts.

[23]  Narendra Ahuja,et al.  Appearance-based eye gaze estimation , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[24]  Oleg V. Komogortsev,et al.  Using machine learning to detect events in eye-tracking data , 2018, Behavior research methods.

[25]  Bernard J. Jansen,et al.  Are personas done? Evaluating their usefulness in the age of digital analytics , 2018, Persona Studies.

[26]  M. Just,et al.  Eye fixations and cognitive processes , 1976, Cognitive Psychology.

[27]  Tim R. H. Cutmore,et al.  Automated eye tracking system calibration using artificial neural networks , 2004, Comput. Methods Programs Biomed..

[28]  Chris Chapman,et al.  The Personas' New Clothes: Methodological and Practical Arguments against a Popular Method , 2006 .

[29]  Jan Stage,et al.  Going Global with Personas , 2013, INTERACT.

[30]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[31]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[32]  Joseph H. Goldberg,et al.  Scanpath clustering and aggregation , 2010, ETRA.

[33]  Helge J. Ritter,et al.  A neural network for 3D gaze recording with binocular eye trackers , 2006, Int. J. Parallel Emergent Distributed Syst..

[34]  Obead Alhadreti,et al.  "Thinking About Thinking Aloud": An Investigation of Think-Aloud Methods in Usability Testing , 2016, BCS HCI.

[35]  Robert P. Sheridan,et al.  Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling , 2003, J. Chem. Inf. Comput. Sci..

[36]  Fumio Mizoguchi,et al.  Evaluation model of cognitive distraction state based on eye-tracking data using neural networks , 2013, 2013 IEEE 12th International Conference on Cognitive Informatics and Cognitive Computing.

[37]  Bryn Wolfe,et al.  A Neural Network Approach to Tracking Eye Position , 1997, Int. J. Hum. Comput. Interact..

[38]  Jie Zhu,et al.  Subpixel eye gaze tracking , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[39]  Stephan Heckers,et al.  Schizophrenia and cognitive function , 2000, Current Opinion in Neurobiology.

[40]  Thomas Ertl,et al.  Triangulating user behavior using eye movement, interaction, and think aloud data , 2016, ETRA.

[41]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[42]  Thierry Baccino,et al.  New insights into ambient and focal visual fixations using an automatic classification algorithm , 2011, i-Perception.

[43]  Yeliz Yesilada,et al.  Scanpath Trend Analysis on Web Pages , 2016, ACM Trans. Web.

[44]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[45]  Keith S. Karn,et al.  Commentary on Section 4. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. , 2003 .

[46]  L. Behera,et al.  A Recurrent Quantum Neural Network Model to Describe Eye Tracking of Moving Targets , 2004, q-bio/0407001.

[47]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[48]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.

[49]  Lori Lorigo,et al.  Eye Monitoring in Online Search , 2008 .

[50]  Bernard J. Jansen,et al.  Fixation and Confusion: Investigating Eye-tracking Participants' Exposure to Information in Personas , 2018, CHIIR.

[51]  Joseph H. Goldberg,et al.  Identifying Aggregate Scanning Strategies to Improve Usability Evaluations , 2010 .

[52]  Bernard J. Jansen,et al.  From 2, 772 segments to five personas: Summarizing a diverse online audience by generating culturally adapted personas , 2018, First Monday.

[53]  Bernard J. Jansen,et al.  Customer segmentation using online platforms: isolating behavioral and demographic segments for persona creation via aggregated user data , 2018, Social Network Analysis and Mining.

[54]  Bernard J. Jansen,et al.  Generating Cultural Personas from Social Data: A Perspective of Middle Eastern Users , 2017, 2017 5th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW).

[55]  Haibo He,et al.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[56]  T. Tenbrink.  Cognitive Discourse Analysis: accessing cognitive representations and processes through language data* , 2014, Language and Cognition.

[57]  Helge J. Ritter,et al.  An Artificial Neural Network for High Precision Eye Movement Tracking , 1994, KI.