EMHMM: Eye Movement Analysis with Hidden Markov Models and Its Applications in Cognitive Research

In many daily life activities, eye movements provide strong clues about underlying cognitive processes. For example, patients with cognitive deficits have atypical eye movement patterns. Users with different experiences show different eye movement behavior in viewing websites. Thus, eye movement has become an important measure in the broad research fields in cognitive science. Recent research has reported substantial individual differences in eye movements during cognitive tasks. Nevertheless, most of the current analysis methods do not adequately reflect these individual differences. Also, they focus on spatial information (fixation locations), whereas temporal information (transitions among fixation locations) is typically overlooked. The most common method has been the use of predefined regions of interests (ROIs) on the stimuli. However, predefined ROIs are often subject to experimenter bias and inconsistency across studies. To address these problems, Caldara and Miellet (2011) proposed to directly perform by-pixel statistical tests on fixation heat maps (where fixations are smoothed with a Gaussian function) to determine the regions with significant difference between conditions. Nevertheless, these regions are often irregularly shaped and difficult to interpret. Also, fixation maps at different times only show the transition of overall fixation distribution and do not provide information about transitions between regions. Another method (Jack et al., 2009) is to define ROIs as regions formed by running the k-means clustering algorithm on significantly fixated regions of a fixation map. However, this approach assumes that all ROIs are circular and the same size, and the number of ROIs must be preset by the experimenter. Thus, we have developed a novel eye movement data analysis method, Eye Movement analysis with Hidden Markov Models (EMHMM; Chuk, Chan, & Hsiao, 2014), which summarizes each individual’s eye movement pattern using a hidden Markov model (HMM; a type of machine learning model for time series data), including person-specific ROIs and transition probabilities among the ROIs. Individual HMMs can be clustered according to similarities to discover common patterns (Fig. 1a), and the similarity between an individual pattern and a common pattern can be quantitatively assessed through estimating the likelihood of the individual’s data being generated by the common pattern HMM. This similarity measure then can be used to examine associations between eye movement patterns and other cognitive measures (Fig. 1b & 1c). We have applied this method to face recognition research and made discoveries thus far not revealed by other methods, including how eye movements are associated with recognition performance, cognitive abilities (Chan, Chan, Lee, & Hsiao, 2018), cultural differences (Chuk, Crookes, et al., 2017), memory encoding/retrieval (Chuk, Chan, & Hsiao, 2017), sleep loss (Zhang, Chan, Lau, & Hsiao, 2019), and activations in brain regions important for top-down attention control (Chan et al., 2016). We have also recently developed new methodologies for more complex cognitive tasks, including using switching HMMs for tasks involving cognitive state changes (Chuk, Chan, Shimojo, & Hsiao, 2016), and using the machine learning algorithm coclustering for tasks involving stimuli with different feature layouts (Hsiao, Chan, Du, & Chan, 2019).