Deep semantic gaze embedding and scanpath comparison for expertise classification during OPT viewing

Modeling eye movement indicative of expertise behavior is decisive in user evaluation. However, it is indisputable that task semantics affect gaze behavior. We present a novel approach to gaze scanpath comparison that incorporates convolutional neural networks (CNN) to process scene information at the fixation level. Image patches linked to respective fixations are used as input for a CNN and the resulting feature vectors provide the temporal and spatial gaze information necessary for scanpath similarity comparison. We evaluated our proposed approach on gaze data from expert and novice dentists interpreting dental radiographs using a local alignment similarity score. Our approach was capable of distinguishing experts from novices with 93% accuracy while incorporating the image semantics. Moreover, our scanpath comparison using image patch features has the potential to incorporate task semantics from a variety of tasks.

[1]  Brent E. Larson,et al.  Visual scan behavior of new and experienced clinicians assessing panoramic radiographs , 2013 .

[2]  James J. Clark,et al.  An inverse Yarbus process: Predicting observers’ task from eye movement patterns , 2014, Vision Research.

[3]  F. Corpet.  Multiple sequence alignment with hierarchical clustering. , 1988, Nucleic acids research.

[4]  Marcus Nyström,et al.  A vector-based, multidimensional scanpath similarity measure , 2010, ETRA.

[5]  C W Douglass,et al.  Clinical efficacy of dental radiography in the detection of dental caries and periodontal diseases. , 1986, Oral surgery, oral medicine, and oral pathology.

[6]  Mei-Ling Shyu,et al.  SP-ASDNet: CNN-LSTM Based ASD Classification Model using Observer ScanPaths , 2019, 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[7]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[8]  E. Lam,et al.  Influence of Experience and Training on Dental Students' Examination Performance Regarding Panoramic Images. , 2016, Journal of dental education.

[9]  Fabian Huettig,et al.  Reporting of dental status from full-arch radiographs: Descriptive analysis and methodological aspects. , 2014, World journal of clinical cases.

[10]  Rong-Fuh Day,et al.  Examining the validity of the Needleman-Wunsch algorithm in identifying decision strategy with eye-movement data , 2010, Decis. Support Syst..

[11]  Katharina Scheiter,et al.  Scanpath comparison in medical image reading skills of dental students: distinguishing stages of expertise development , 2018, ETRA.

[12]  L. Itti,et al.  Defending Yarbus: eye movements reveal observers' task. , 2014, Journal of vision.

[13]  R. Säljö,et al.  Expertise Differences in the Comprehension of Visualizations: a Meta-Analysis of Eye-Tracking Research in Professional Domains , 2011 .

[14]  Tianming Liu,et al.  Predicting eye fixations using convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Michelle R. Greene,et al.  Reconsidering Yarbus: A failure to predict observers’ task from eye movement patterns , 2012, Vision Research.

[16]  Katharina Scheiter,et al.  Overlooking: the nature of gaze behavior and anomaly detection in expert dentists , 2018, MCPMD@ICMI.

[17]  Thiago Santini,et al.  Encodji: encoding gaze data into emoji space for an amusing scanpath classification approach ;) , 2019, ETRA.

[18]  Andrew Begel,et al.  Eye Movements in Code Reading: Relaxing the Linear Order , 2015, 2015 IEEE 23rd International Conference on Program Comprehension.

[19]  Wolfgang Rosenstiel,et al.  SubsMatch 2.0: Scanpath comparison and classification based on subsequence frequencies , 2016, Behavior Research Methods.

[20]  C. J. Ravesloot,et al.  How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology , 2017, Advances in health sciences education : theory and practice.

[21]  Noel E. O'Connor,et al.  SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[22]  Joseph H. Goldberg,et al.  Scanpath clustering and aggregation , 2010, ETRA.

[23]  Matthias Bethge,et al.  Information-theoretic model comparison unifies saliency metrics , 2015, Proceedings of the National Academy of Sciences.

[24]  Thies Pfeiffer,et al.  EyeSee3D 2.0: model-based real-time analysis of mobile eye-tracking in static and dynamic three-dimensional scenes , 2016, ETRA.

[25]  Qi Zhao,et al.  SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[26]  Jan Theeuwes,et al.  ScanMatch: A novel method for comparing fixation sequences , 2010, Behavior research methods.

[27]  Christus,et al.  A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins , 2022 .

[28]  David J. Heeger,et al.  Corrigendum: Analysis of Perceptual Expertise in Radiology – Current Knowledge and a New Perspective , 2019, Front. Hum. Neurosci..

[29]  Andrea Kienle,et al.  The Influence of Different AOI Models in Source Code Comprehension Analysis , 2019, 2019 IEEE/ACM 6th International Workshop on Eye Movements in Programming (EMIP).

[30]  Pushpak Bhattacharyya,et al.  Automatic Extraction of Cognitive Features from Gaze Data , 2018 .

[31]  Garrison W. Cottrell,et al.  Predicting an observer's task using multi-fixation pattern analysis , 2014, ETRA.

[32]  Wolfgang Rosenstiel,et al.  SubsMatch: scanpath similarity in dynamic scenes based on subsequence frequencies , 2014, ETRA.

[33]  Z Z Akarslan,et al.  A comparison of the diagnostic accuracy of bitewing, periapical, unfiltered and filtered digital panoramic images for approximal caries detection in posterior teeth. , 2008, Dento maxillo facial radiology.

[34]  Yiwen Sun,et al.  Automatic analysis of eye tracking data for medical diagnosis , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  M S Waterman,et al.  Identification of common molecular subsequences. , 1981, Journal of molecular biology.

[37]  Michael Burch,et al.  EyeMSA: exploring eye movement data with pairwise and multiple sequence alignment , 2018, ETRA.

[38]  Anne R. Haake,et al.  eyePatterns: software for identifying patterns and similarities across fixation sequences , 2006, ETRA.

[39]  S. C. Johnson.  Hierarchical clustering schemes , 1967, Psychometrika.

[40]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Arzu Çöltekin,et al.  Exploring the efficiency of users' visual analytics strategies based on sequence analysis of eye movement recordings , 2010, Int. J. Geogr. Inf. Sci..

[42]  Christos A. Ouzounis,et al.  Computational complexity of algorithms for sequence comparison, short-read assembly and genome alignment , 2017, Biosyst..

[43]  Huchuan Lu,et al.  Deep networks for saliency detection via local estimation and global search , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  K. Scheiter,et al.  Conveying clinical reasoning based on visual observation via eye-movement modelling examples , 2012, Instructional Science.

[45]  Caroline Jay,et al.  Exploring the Relationship Between Eye Movements and Electrocardiogram Interpretation Accuracy , 2016, Scientific Reports.

[46]  M Gelfand,et al.  Reliability of radiographical interpretations. , 1983, Journal of endodontics.

[47]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Anna Rozenshtein,et al.  Effect of Massed Versus Interleaved Teaching Method on Performance of Students in Radiology. , 2016, Journal of the American College of Radiology : JACR.

[49]  Seunghoon Hong,et al.  Online Tracking by Learning Discriminative Saliency Map with Convolutional Neural Network , 2015, ICML.

[50]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[51]  Alan Kingstone,et al.  A comparison of scanpath comparison methods , 2014, Behavior Research Methods.

[52]  Thierry Baccino,et al.  Methods for comparing scanpaths and saliency maps: strengths and weaknesses , 2012, Behavior Research Methods.

[53]  David J. Heeger,et al.  Analysis of Perceptual Expertise in Radiology – Current Knowledge and a New Perspective , 2019, Front. Hum. Neurosci..

[54]  Halszka Jarodzka,et al.  Systematic viewing in radiology: seeing more, missing less? , 2016, Advances in health sciences education : theory and practice.

[55]  M. Stella Atkins,et al.  Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment , 2004, ETRA.

[56]  Michael A. Bruno,et al.  Understanding and Confronting Our Mistakes: The Epidemiology of Error in Radiology and Strategies for Error Reduction. , 2015, Radiographics : a review publication of the Radiological Society of North America, Inc.

[57]  Alan Kennedy,et al.  Book Review: Eye Tracking: A Comprehensive Guide to Methods and Measures , 2016, Quarterly journal of experimental psychology.

[58]  Srijith Rajeev,et al.  Fixation oriented object segmentation using mobile eye tracker , 2018, Commercial + Scientific Sensing and Imaging.

[59]  T. Gog,et al.  How to Convey Perceptual Skills by Displaying Experts’ Gaze Data , 2009 .

[60]  Claude Frasson,et al.  Local Sequence Alignment for Scan Path Similarity Assessment , 2018 .