Tracking Story Reading in the Brain

Story comprehension is a rich and rapid phenomenon that requires multiple simultaneous processes (e.g. letter recognition, word understanding, sentence parsing...). Our goal is to study these complex processes in the brain by modeling the fMRI brain activity during story reading at a close to normal speed. This is a challenging goal, one reason being the coarse time-resolution of fMRI and another the lack of a comprehensive model of word meaning composition. Classically, fMRI has been used to localize brain areas that process specific elements of text processing (e.g. which areas are involved in syntactic processing) but not to model how the brain represents different instances of these elements (e.g. how do those areas represent different syntactic structures). We present here a generative model that predicts the fMRI activity created when subjects read a complex story where the words are presented in a serial manner, for 0.5 seconds each. Using this model, we performed an exploratory analysis in which we tested several types of story features (e.g. word length, syntax, semantics, story characters) to search for a good basis of features for story comprehension. We found different patterns of representation in the brain for different types of features. These patterns align with the predictions from the field. We tested the expressivity of our model using a classification task that decodes a passage of the story from a time segment of brain activity. We obtain a classification accuracy that is significantly higher than chance with p < 10−6. We show that we can indeed study multiple components of reading simultaneously in fMRI at a close to normal speed. Our approach has the advantage of being flexible: any feature of language can be added to the model and tested, and features can range from simple perceptual features, to compositional semantics, to higher order reasoning about narrative structure and story comprehension.