Advanced statistical methods for eye movement analysis and modeling: a gentle introduction

In this Chapter we consider eye movements and, in particular, the resulting sequence of gaze shifts to be the observable outcome of a stochastic process. Crucially, we show that, under such assumption, a wide variety of tools become available for analyses and modelling beyond conventional statistical methods. Such tools encompass random walk analyses and more complex techniques borrowed from the Pattern Recognition and Machine Learning fields. After a brief, though critical, probabilistic tour of current computational models of eye movements and visual attention, we lay down the basis for gaze shift pattern analysis. To this end, the concepts of Markov Processes, the Wiener process and related random walks within the Gaussian framework of the Central Limit Theorem will be introduced. Then, we will deliberately violate fundamental assumptions of the Central Limit Theorem to elicit a larger perspective, rooted in statistical physics, for analysing and modelling eye movements in terms of anomalous, non-Gaussian, random walks and modern foraging theory. Eventually, by resorting to Statistical Machine Learning techniques, we discuss how the analyses of movement patterns can develop into the inference of hidden patterns of the mind: inferring the observer’s task, assessing cognitive impairments, classifying expertise.

[1]  T. Foulsham,et al.  What can saliency models predict about eye movements? Spatial and sequential aspects of fixations during encoding and recognition. , 2008, Journal of vision.

[2]  George J. Stigler,et al.  Public Regulation of the Securities Markets , 1964 .

[3]  Alexander C. Schütz,et al.  Eye movements and perception: a selective review. , 2011, Journal of vision.

[4]  C. Koch,et al.  Faces and text attract gaze independent of the task: Experimental data and computer model. , 2009, Journal of vision.

[5]  Edward Nelson Dynamical Theories of Brownian Motion , 1967 .

[6]  G. Zelinsky A theory of eye movements during target acquisition. , 2008, Psychological review.

[7]  G. Humphreys,et al.  Attention, spatial representation, and visual neglect: simulating emergent attention and spatial memory in the selective attention for identification model (SAIM). , 2003, Psychological review.

[8]  R. Mazo On the theory of brownian motion , 1973 .

[9]  Tohru Ozaki,et al.  Time Series Modeling of Neuroscience Data , 2012 .

[10]  B. Gnedenko,et al.  Limit Distributions for Sums of Independent Random Variables , 1955 .

[11]  C. Mallows,et al.  A Method for Simulating Stable Random Variables , 1976 .

[12]  Rajesh P. N. Rao,et al.  Eye movements in iconic visual search , 2002, Vision Research.

[13]  M. Newman Power laws, Pareto distributions and Zipf's law , 2005 .

[14]  John M. Henderson,et al.  Predicting Cognitive State from Eye Movements , 2013, PloS one.

[15]  Benjamin W. Tatler,et al.  Systematic tendencies in scene viewing , 2008 .

[16]  R. Friedrich,et al.  Modeling of nonlinear Lévy processes by data analysis. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[17]  Michael P. Wiper,et al.  Bayesian Analysis of Stochastic Process Models: Ruggeri/Bayesian Analysis of Stochastic Process Models , 2012 .

[18]  Kunio Kashino,et al.  Dynamic Markov random fields for stochastic modeling of visual attention , 2008, 2008 19th International Conference on Pattern Recognition.

[19]  P. Cowpertwait,et al.  Introductory Time Series with R , 2009 .

[20]  Ralf Engbert,et al.  Bayesian estimation of the scaling parameter of fixational eye movements , 2012 .

[21]  Ralf Engbert Microsaccades: A microcosm for research on oculomotor control, attention, and visual perception. , 2006, Progress in brain research.

[22]  H. Stanley,et al.  The Physics of Foraging: An Introduction to Random Searches and Biological Encounters , 2011 .

[23]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[24]  M. Osborne Brownian Motion in the Stock Market , 1959 .

[25]  Theodoros Damoulas,et al.  Pattern Recognition , 1998, Encyclopedia of Information Systems.

[26]  Michael P. Wiper,et al.  Bayesian Analysis of Stochastic Process Models , 2012 .

[27]  Filippo Radicchi,et al.  Levy flights in human behavior and cognition , 2013, 1306.6533.

[28]  Josep Lladós,et al.  Modelling Task-Dependent Eye Guidance to Objects in Pictures , 2014, Cognitive Computation.

[29]  S. Martinez-Conde,et al.  An oculomotor continuum from exploration to fixation , 2013, Proceedings of the National Academy of Sciences.

[30]  Theodoros Damoulas,et al.  Multiclass Relevance Vector Machines: Sparsity and Accuracy , 2010, IEEE Transactions on Neural Networks.

[31]  M. Tribus,et al.  Probability theory: the logic of science , 2003 .

[32]  Junchi Yan,et al.  Visual Saliency Detection via Sparsity Pursuit , 2010, IEEE Signal Processing Letters.

[33]  S. Benhamou HOW MANY ANIMALS REALLY DO THE LÉVY WALK , 2007 .

[34]  Naila Murray,et al.  End-to-End Saliency Mapping via Probability Distribution Prediction , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Benjamin W Tatler,et al.  The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. , 2007, Journal of vision.

[36]  Paolo Napoletano,et al.  What the Draughtsman's Hand Tells the Draughtsman's Eye: a Sensorimotor Account of Drawing , 2008, Int. J. Pattern Recognit. Artif. Intell..

[37]  Bernhard Schölkopf,et al.  Center-surround patterns emerge as optimal predictors for human saccade targets. , 2009, Journal of vision.

[38]  A. Einstein Über die von der molekularkinetischen Theorie der Wärme geforderte Bewegung von in ruhenden Flüssigkeiten suspendierten Teilchen [AdP 17, 549 (1905)] , 2005, Annalen der Physik.

[39]  T. Poggio,et al.  What and where: A Bayesian inference theory of attention , 2010, Vision Research.

[40]  Jochen J. Steil,et al.  Where to Look Next? Combining Static and Dynamic Proto-objects in a TVA-based Model of Visual Attention , 2010, Cognitive Computation.

[41]  Antoine Coutrot,et al.  Face exploration dynamics differentiate men and women. , 2016, Journal of vision.

[42]  C. Gardiner Stochastic Methods: A Handbook for the Natural and Social Sciences , 2009 .

[43]  Edward A. Codling,et al.  Random walk models in biology , 2008, Journal of The Royal Society Interface.

[44]  Giuseppe Boccignone,et al.  Gaze shift behavior on video as composite information foraging , 2013, Signal Process. Image Commun..

[45]  Giuseppe Boccignone,et al.  Ecological Sampling of Gaze Shifts , 2014, IEEE Transactions on Cybernetics.

[46]  A. Einstein On the movement of small particles suspended in a stationary liquid demanded by the molecular-kinetic theory of heart , 1905 .

[47]  Todd S. Horowitz,et al.  Visual search has no memory , 1998, Nature.

[48]  J ValdésJulio,et al.  2006 Special issue , 2006 .

[49]  Alexander H. Hartmann,et al.  Introduction to Statistical Physics , 2003 .

[50]  Deborah J. Aks,et al.  Memory Across Eye-Movements: 1/f Dynamic in Visual Search , 2010 .

[51]  G. Viswanathan,et al.  Lévy flights and superdiffusion in the context of biological encounters and random searches , 2008 .

[52]  G. Humphreys,et al.  Computational models of visual selective attention: A review , 2005 .

[53]  Francesco Tisato,et al.  Attentive Monitoring of Multiple Video Streams Driven by a Bayesian Foraging Strategy , 2014, IEEE Transactions on Image Processing.

[54]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[55]  Mozer,et al.  Early parallel processing in reading: a connectionist approach. Technical report, April-November 1986 , 1986 .

[56]  Ioannis A. Koutrouvelis,et al.  Regression-Type Estimation of the Parameters of Stable Laws , 1980 .

[57]  Christof Koch,et al.  Modeling attention to salient proto-objects , 2006, Neural Networks.

[58]  Roger Ratcliff,et al.  The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks , 2008, Neural Computation.

[59]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[60]  F. M. D. P. Martín,et al.  A Theory of Reaction Time Distributions , 2009 .

[61]  B. Mandelbrot,et al.  Fractional Brownian Motions, Fractional Noises and Applications , 1968 .

[62]  Qiang Ji,et al.  Deep eye fixation map learning for calibration-free eye gaze tracking , 2016, ETRA.

[63]  Bernardo Spagnolo,et al.  Lévy Flight Superdiffusion: an Introduction , 2008, Int. J. Bifurc. Chaos.

[64]  R. H. Phaf,et al.  SLAM: A connectionist model for attention in visual selection tasks , 1990, Cognitive Psychology.

[65]  B. Scholl Objects and attention: the state of the art , 2001, Cognition.

[66]  P. Perona,et al.  Objects predict fixations better than early saliency. , 2008, Journal of vision.

[67]  D. Ballard,et al.  Eye guidance in natural vision: reinterpreting salience. , 2011, Journal of vision.

[68]  Yueting Zhuang,et al.  Saliency Detection within a Deep Convolutional Architecture , 2014, AAAI 2014.

[69]  Christof Koch,et al.  Predicting human gaze using low-level saliency combined with face detection , 2007, NIPS.

[70]  Eugene Agichtein,et al.  Detecting cognitive impairment by eye movement analysis using automatic classification algorithms , 2011, Journal of Neuroscience Methods.

[71]  Robert B. Fisher,et al.  A computer vision model for visual-object-based attention and eye movements , 2008, Comput. Vis. Image Underst..

[72]  Gary Feng,et al.  Eye movements as time-series random variables: A stochastic model of eye movement control in reading , 2006, Cognitive Systems Research.

[73]  Daniel Campos,et al.  Stochastic Foundations in Movement Ecology , 2014 .

[74]  P. Meyer Stochastic Processes from 1950 to the Present , 2009 .

[75]  Wilson S. Geisler,et al.  Optimal eye movement strategies in visual search , 2005, Nature.

[76]  W. Ebeling Stochastic Processes in Physics and Chemistry , 1995 .

[77]  Desmond J. Higham,et al.  An Algorithmic Introduction to Numerical Simulation of Stochastic Differential Equations , 2001, SIAM Rev..

[78]  R. Desimone,et al.  Neural mechanisms of selective visual attention. , 1995, Annual review of neuroscience.

[79]  Andy M. Reynolds,et al.  Optimal random Lévy-loop searching: New insights into the searching behaviours of central-place foragers , 2008 .

[80]  Matthias Bethge,et al.  Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet , 2014, ICLR.

[81]  C. Schinckus How Physicists Made Stable Lévy Processes Physically Plausible , 2013 .

[82]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[83]  Dietmar Heinke,et al.  Modelling Visual Search with the Selective Attention for Identification Model (VS-SAIM): A Novel Explanation for Visual Search Asymmetries , 2010, Cognitive Computation.

[84]  Lawrence W. Stark,et al.  Top-down guided eye movements , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[85]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[86]  Peyman Milanfar,et al.  Static and space-time visual saliency detection by self-resemblance. , 2009, Journal of vision.

[87]  K. Lau,et al.  On generalized harmonic analysis , 1980 .

[88]  A Treisman,et al.  Feature binding, attention and object perception. , 1998, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[89]  James J. Clark,et al.  A computational model for task inference in visual search. , 2013, Journal of vision.

[90]  Daniel Campos,et al.  Stochastic Foundations in Movement Ecology: Anomalous Diffusion, Front Propagation and Random Searches , 2013 .

[91]  R. Baddeley,et al.  The long and the short of it: Spatial statistics at fixation vary with saccade amplitude and task , 2006, Vision Research.

[92]  W. Steubing,et al.  Zur Theorie der Brownschen Bewegung , 1908 .

[93]  O. Meur,et al.  Introducing context-dependent and spatially-variant viewing biases in saccadic models , 2016, Vision Research.

[94]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[95]  David Barber,et al.  Bayesian Time Series Models , 2011 .

[96]  Arnold W. M. Smeulders,et al.  Brain responses strongly correlate with Weibull image statistics when processing natural images. , 2009, Journal of vision.

[97]  M. Plank,et al.  Optimal foraging: Lévy pattern or process? , 2008, Journal of The Royal Society Interface.

[98]  Yihua Tan,et al.  Maximal Entropy Random Walk for Region-Based Visual Saliency , 2014, IEEE Transactions on Cybernetics.

[99]  Antonio Torralba,et al.  Contextual Priming for Object Detection , 2003, International Journal of Computer Vision.

[100]  Giuseppe Boccignone,et al.  The Active Sampling of Gaze-Shifts , 2011, ICIAP.

[101]  Henrik I. Christensen,et al.  Computational visual attention systems and their cognitive foundations: A survey , 2010, TAP.

[102]  Rosario N. Mantegna,et al.  Book Review: An Introduction to Econophysics, Correlations, and Complexity in Finance, N. Rosario, H. Mantegna, and H. E. Stanley, Cambridge University Press, Cambridge, 2000. , 2000 .

[103]  L. Bachelier,et al.  Théorie de la spéculation , 1900 .

[104]  Bernhard Schölkopf,et al.  A Nonparametric Approach to Bottom-Up Visual Saliency , 2006, NIPS.

[105]  Nathalie Guyader,et al.  Improving Visual Saliency by Adding ‘Face Feature Map’ and ‘Center Bias’ , 2012, Cognitive Computation.

[106]  Cristian Sminchisescu,et al.  Action from Still Image Dataset and Inverse Optimal Control to Learn Task Specific Visual Scanpaths , 2013, NIPS.

[107]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[108]  G. Logan The CODE theory of visual attention: an integration of space-based and object-based attention. , 1996, Psychological review.

[109]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[110]  Paolo Napoletano,et al.  Bayesian Integration of Face and Low-Level Cues for Foveated Video Coding , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[111]  J. Doob,et al.  The Brownian Movement and Stochastic Equations , 1942 .

[112]  P. Trillenberg,et al.  Random walks, random sequences, and nonlinear dynamics in human optokinetic nystagmus. , 2001, Journal of applied physiology.

[113]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[114]  C. Geiss,et al.  An introduction to probability theory , 2008 .

[115]  S. Martinez-Conde,et al.  The impact of microsaccades on vision: towards a unified theory of saccadic function , 2013, Nature Reviews Neuroscience.

[116]  H. Larralde,et al.  Lévy walk patterns in the foraging movements of spider monkeys (Ateles geoffroyi) , 2003, Behavioral Ecology and Sociobiology.

[117]  Zulin Wang,et al.  Image Saliency Detection with Sparse Representation of Learnt Texture Atoms , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[118]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[119]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[120]  O. Schwartz,et al.  Visuomotor characterization of eye movements in a drawing task , 2009, Vision Research.

[121]  B. Mandlebrot The Variation of Certain Speculative Prices , 1963 .

[122]  Guido C. H. E. de Croon,et al.  Adaptive Gaze Control for Object Detection , 2011, Cognitive Computation.

[123]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[124]  C Bundesen,et al.  A computational theory of visual attention. , 1998, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[125]  Simon Rogers,et al.  A First Course in Machine Learning , 2011, Chapman and Hall / CRC machine learning and pattern recognition series.

[126]  Theo Geisel,et al.  The ecology of gaze shifts , 2000, Neurocomputing.

[127]  Cristian Sminchisescu,et al.  Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[128]  Nathalie Guyader,et al.  A Computational Saliency Model Integrating Saccade Programming , 2009, BIOSIGNALS.

[129]  M. Ferraro,et al.  The Foraging Brain: Evidence of Lévy Dynamics in Brain Networks , 2016, bioRxiv.

[130]  L. Richardson,et al.  Atmospheric Diffusion Shown on a Distance-Neighbour Graph , 1926 .

[131]  Qi Zhao,et al.  Learning to predict eye fixations for semantic contents using multi-layer sparse network , 2014, Neurocomputing.

[132]  Alessandro Chessa,et al.  A Statistical Physics Perspective to Understand Social Visual Attention in Autism Spectrum Disorder , 2017, Perception.

[133]  Philip Protter,et al.  A short history of stochastic integration and mathematical finance the early years, 1880-1970 , 2004 .

[134]  Preeti Verghese,et al.  The psychophysics of visual search , 2000, Vision Research.

[135]  H. J. Muller,et al.  SEarch via Recursive Rejection (SERR): A Connectionist Model of Visual Search , 1993, Cognitive Psychology.

[136]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[137]  R. H. S. Carpenter,et al.  The LATER model of reaction time and decision , 2016, Neuroscience & Biobehavioral Reviews.

[138]  C. Koch,et al.  Probabilistic modeling of eye movement data during conjunction search via feature-based attention. , 2007, Journal of vision.

[139]  Laurent Itti,et al.  A Bayesian model for efficient visual search and recognition , 2010, Vision Research.

[140]  R. Venkatesh Babu,et al.  DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations , 2015, IEEE Transactions on Image Processing.

[141]  J. L. Nolan,et al.  Numerical calculation of stable densities and distribution functions: Heavy tails and highly volatil , 1997 .

[142]  Ralf Engbert,et al.  An integrated model of fixational eye movements and microsaccades , 2011, Proceedings of the National Academy of Sciences.

[143]  Ravi Kiran Sarvadevabhatla,et al.  A Taxonomy of Deep Convolutional Neural Nets for Computer Vision , 2016, Front. Robot. AI.

[144]  Ali Borji,et al.  An Object-Based Bayesian Framework for Top-Down Visual Attention , 2012, AAAI.

[145]  P. Levy Théorie de l'addition des variables aléatoires , 1955 .

[146]  Fakhri Karray,et al.  A Probabilistic Model of Overt Visual Attention for Cognitive Robots , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[147]  Lawrence W. Stark,et al.  Visual perception and sequences of eye movement fixations: a stochastic modeling approach , 1992, IEEE Trans. Syst. Man Cybern..

[148]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[149]  Giuseppe Boccignone,et al.  Feed and fly control of visual scanpaths for foveation image processing , 2012, annals of telecommunications - annales des télécommunications.

[150]  B. Tatler,et al.  The prominence of behavioural biases in eye guidance , 2009 .

[151]  Edward Vul,et al.  A Bayesian Optimal Foraging Model of Human Visual Search , 2012, Psychological science.

[152]  R. H. S. Carpenter,et al.  Neural computation of log likelihood in control of saccadic eye movements , 1995, Nature.

[153]  J. Wolfe,et al.  Guided Search 2.0 A revised model of visual search , 1994, Psychonomic bulletin & review.

[154]  Terry Caelli,et al.  An information-theoretic approach to active vision , 2001, Proceedings 11th International Conference on Image Analysis and Processing.

[155]  A. Destexhe,et al.  Stochastic Methods in Neuroscience , 2012 .

[156]  Roxanne L. Canosa,et al.  Real-world vision: Selective perception and task , 2009, TAP.

[157]  Jian Yu,et al.  Saliency Detection by Multitask Sparsity Pursuit , 2012, IEEE Transactions on Image Processing.

[158]  Andy Reynolds,et al.  How many animals really do the Lévy walk? Comment. , 2008, Ecology.

[159]  Refractor Vision , 2000, The Lancet.

[160]  C. Erkelens,et al.  Coarse-to-fine eye movement strategy in visual search , 2007, Vision Research.

[161]  V. M. Tikhomirov,et al.  Dissipation of Energy in Isotropic Turbulence , 1991 .

[162]  Jörg Baschnagel,et al.  Stochastic Processes: From Physics to Finance , 2000 .

[163]  T D Keech,et al.  Eye movements in active visual search: A computable phenomenological model , 2010, Attention, perception & psychophysics.

[164]  Jeremy M Wolfe,et al.  When is it time to move to the next raspberry bush? Foraging rules in human visual search. , 2013, Journal of vision.

[165]  Mario Ferraro,et al.  Detecting expert’s eye using a multiple-kernel Relevance Vector Machine , 2014 .

[166]  Stefano Soatto,et al.  Deformotion: Deforming Motion, Shape Average and the Joint Registration and Approximation of Structures in Images , 2003, International Journal of Computer Vision.

[167]  Michael Dorr,et al.  Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[168]  A. L. I︠A︡rbus Eye Movements and Vision , 1967 .

[169]  S R Ellis,et al.  Statistical Dependency in Visual Scanning , 1986, Human factors.

[170]  Daniel Mirman,et al.  Lévy-like diffusion in eye movements during spoken-language comprehension. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[171]  David Marr,et al.  VISION A Computational Investigation into the Human Representation and Processing of Visual Information , 2009 .

[172]  Ronald A. Rensink The Dynamic Representation of Scenes , 2000 .

[173]  Umesh Rajashekar,et al.  DOVES: a database of visual eye movements. , 2009, Spatial vision.

[174]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[175]  Giuseppe Boccignone,et al.  Modelling gaze shift as a constrained random walk , 2004 .

[176]  Thomas T. Hills Animal Foraging and the Evolution of Goal-Directed Cognition , 2006, Cogn. Sci..

[177]  Scott Cheng-Hsin Yang,et al.  Theoretical perspectives on active sensing , 2016, Current Opinion in Behavioral Sciences.

[178]  Michael Bach,et al.  Object perception: When our brain is impressed but we do not notice it , 2009 .