Episodes, events, and models

We describe a novel computational theory of how individuals segment perceptual information into representations of events. The theory is inspired by recent findings in the cognitive science and cognitive neuroscience of event segmentation. In line with recent theories, it holds that online event segmentation is automatic, and that event segmentation yields mental simulations of events. But it posits two novel principles as well: first, discrete episodic markers track perceptual and conceptual changes, and can be retrieved to construct event models. Second, the process of retrieving and reconstructing those episodic markers is constrained and prioritized. We describe a computational implementation of the theory, as well as a robotic extension of the theory that demonstrates the processes of online event segmentation and event model construction. The theory is the first unified computational account of event segmentation and temporal inference. We conclude by demonstrating now neuroimaging data can constrain and inspire the construction of process-level theories of human reasoning.

[1]  P. Johnson-Laird,et al.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness , 1985 .

[2]  Jonathan Berger,et al.  Neural Dynamics of Event Segmentation in Music: Converging Evidence for Dissociable Ventral and Dorsal Networks , 2007, Neuron.

[3]  Jeffrey M. Zacks,et al.  Event perception: a mind-brain perspective. , 2007, Psychological bulletin.

[4]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[5]  Vinod Goel,et al.  Anatomy of deductive reasoning , 2007, Trends in Cognitive Sciences.

[6]  G. Rizzolatti,et al.  Congruent Embodied Representations for Visually Presented Actions and Linguistic Phrases Describing Actions , 2006, Current Biology.

[7]  William F. Brewer,et al.  Memory for goal-directed events , 1980, Cognitive Psychology.

[8]  M. Witter,et al.  A Specific Role of the Human Hippocampus in Recall of Temporal Sequences , 2009, The Journal of Neuroscience.

[9]  William F. Brewer,et al.  Use of plan schemata in the recall and recognition of goal-directed actions. , 1983 .

[10]  Jeffrey M. Zacks,et al.  Parsing activity into meaningful events , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[11]  M. Zorzi,et al.  When time is space: Evidence for a mental time line , 2012, Neuroscience & Biobehavioral Reviews.

[12]  H. Minkowski,et al.  Space and time , 1952 .

[13]  Chantal E Stern,et al.  The retrieval of learned sequences engages the hippocampus: Evidence from fMRI , 2009, Hippocampus.

[14]  R. Byrne Précis of The Rational Imagination: How People Create Alternatives to Reality , 2007, Behavioral and Brain Sciences.

[15]  T. Harley Does Cognitive Neuropsychology have a future? , 2004, Cognitive neuropsychology.

[16]  Jeffrey M. Zacks,et al.  Reading Stories Activates Neural Representations of Visual and Motor Experiences , 2009, Psychological science.

[17]  P. Johnson-Laird,et al.  Mental models and temporal reasoning , 1996, Cognition.

[18]  G. Bower Plans and Goals in Understanding Episodes , 1982 .

[19]  Melissa J. Allman,et al.  Properties of the internal clock: first- and second-order principles of subjective time. , 2014, Annual review of psychology.

[20]  B. Staresina,et al.  Mind the Gap: Binding Experiences across Space and Time in the Human Hippocampus , 2009, Neuron.

[21]  Jeffrey M. Zacks,et al.  Perceiving, remembering, and communicating structure in events. , 2001, Journal of experimental psychology. General.

[22]  A. Vandierendonck,et al.  Working Memory Constraints on Linear Reasoning with Spatial and Temporal Contents , 1997, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[23]  Jeffrey M. Zacks,et al.  The Brain's Cutting-Room Floor: Segmentation of Narrative Cinema , 2010, Front. Hum. Neurosci..

[24]  C. Peirce,et al.  Collected Papers of Charles Sanders Peirce , 1936, Nature.

[25]  Daniel Casasanto,et al.  Space and Time in the Child's Mind: Evidence for a Cross-Dimensional Asymmetry , 2010, Cogn. Sci..

[26]  Darren Newtson Attribution and the unit of perception of ongoing behavior. , 1973 .

[27]  Achille C. Varzi,et al.  Event Concepts , 2007 .

[28]  M. Coltheart What has Functional Neuroimaging told us about the Mind (so far)? (Position Paper Presented to the European Cognitive Neuropsychology Workshop, Bressanone, 2005) , 2006, Cortex.

[29]  Darren Newtson,et al.  The objective basis of behavior units. , 1977 .

[30]  A. Vandierendonck,et al.  Plugging a tooth before anaesthetising the patient? The influence of people's beliefs on reasoning about the temporal order of actions , 2004 .

[31]  Evan Heit,et al.  Brain Imaging, Forward Inference, and Theories of Reasoning , 2015, Front. Hum. Neurosci..

[32]  Jeffrey M. Zacks,et al.  Understanding events : from perception to action , 2008 .

[33]  Panos E. Trahanias,et al.  Temporal Cognition: A Key Ingredient of Intelligent Systems , 2011, Front. Neurorobot..

[34]  W. Uttal Mind and Brain: A Critical Appraisal of Cognitive Neuroscience , 2011 .

[35]  B. Tversky,et al.  Making sense of abstract events: Building event schemas , 2006, Memory & cognition.

[36]  Gabriel A Radvansky,et al.  Walking through doorways causes forgetting: Situation models and experienced space , 2006, Memory & cognition.

[37]  L. Harner Yesterday and Tomorrow: Development of Early Understanding of the Terms , 1975 .

[38]  Judith A. Hudson,et al.  From knowing to telling: The development of children's scripts, stories, and personal narratives. , 1991 .

[39]  T. Shallice,et al.  Hierarchical schemas and goals in the control of sequential behavior. , 2006, Psychological review.

[40]  Franziska M. Korb,et al.  The fraction of an action is more than a movement: Neural signatures of event segmentation in fMRI , 2012, NeuroImage.

[41]  Markus Knauff,et al.  A theory and a computational model of spatial reasoning with preferred mental models. , 2013, Psychological review.

[42]  Gabriel A Radvansky,et al.  Walking through doorways causes forgetting: Environmental integration , 2010, Psychonomic bulletin & review.

[43]  Jeffrey M. Zacks,et al.  Human brain activity time-locked to perceptual event boundaries , 2001, Nature Neuroscience.

[44]  Jeffrey M. Zacks,et al.  Event perception , 2011, Scholarpedia.

[45]  Elliot M. Meyerowitz,et al.  MscS-like Proteins Control Plastid Size and Shape in Arabidopsis thaliana , 2006, Current Biology.

[46]  Emilio Kropff,et al.  Place cells, grid cells, and the brain's spatial representation system. , 2008, Annual review of neuroscience.

[47]  Brandon M. Liverence,et al.  Discrete events as units of perceived time. , 2012, Journal of experimental psychology. Human perception and performance.

[48]  Raj M. Ratwani,et al.  A memory for goals model of sequence errors , 2011, Cognitive Systems Research.

[49]  edrt CJentner Spatial Metaphors in Temporal Reasoning , 2003 .

[50]  M. Botvinick,et al.  Neural representations of events arise from temporal community structure , 2013, Nature Neuroscience.

[51]  James R. Booth,et al.  The Brain Network for Deductive Reasoning: A Quantitative Meta-analysis of 28 Neuroimaging Studies , 2011, Journal of Cognitive Neuroscience.

[52]  D. Osherson,et al.  The boundaries of language and thought in deductive inference , 2009, Proceedings of the National Academy of Sciences.

[53]  T. Baguley,et al.  Long-Term Memory for Spatial and Temporal Mental Models Includes Construction Processes and Model Structure , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[54]  L. Davachi,et al.  What Constitutes an Episode in Episodic Memory? , 2011, Psychological science.

[55]  H. Eichenbaum,et al.  Hippocampal “Time Cells” Bridge the Gap in Memory for Discontiguous Events , 2011, Neuron.

[56]  J. Gregory Trafton,et al.  Leveraging Cognitive Context for Object Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[57]  Behrooz Kamgar-Parsi,et al.  Toward Development of a Face Recognition System for Watchlist Surveillance , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Jonathan D. Cohen,et al.  Distinct neural substrates for deductive and mathematical processing , 2008, Brain Research.

[59]  Mark Steedman,et al.  Temporal Ontology and Temporal Reference , 1988, CL.

[60]  Jeffery. M. Zacks,et al.  Activation of human motion processing areas during event perception , 2003, Cognitive, affective & behavioral neuroscience.

[61]  Mike Oaksford,et al.  Imaging deductive reasoning and the new paradigm , 2015, Front. Hum. Neurosci..

[62]  J. Gregory Trafton,et al.  Memory for goals: an activation-based model , 2002, Cogn. Sci..

[63]  K. Nelson,et al.  Event knowledge : structure and function in development , 1986 .

[64]  J. Gregory Trafton,et al.  Embodied Spatial Cognition , 2011, Top. Cogn. Sci..

[65]  Susan Bell Trickett,et al.  "What if...": The Use of Conceptual Simulations in Scientific Reasoning , 2007, Cogn. Sci..

[66]  Stephen J. Anderson,et al.  Investigating the structure of autobiographical memories. , 1993 .

[67]  John-Dylan Haynes,et al.  Distributed Representations of Rule Identity and Rule Order in Human Frontal Cortex and Striatum , 2012, The Journal of Neuroscience.

[68]  J. Brewer,et al.  Activity in the hippocampus and neocortical working memory regions predicts successful associative memory for temporally discontiguous events , 2010, Neuropsychologia.

[69]  Sangeet Khemlani,et al.  Toward a Unified Theory of Reasoning , 2013 .

[70]  D. Kumaran,et al.  The Dynamics of Hippocampal Activation during Encoding of Overlapping Sequences , 2006, Neuron.

[71]  P N Johnson-Laird,et al.  Reasoning about relations. , 2005, Psychological review.

[72]  J. Gregory Trafton,et al.  ACT-R/E , 2013, HRI 2013.

[73]  Karen Wynn,et al.  Infants' Individuation and Enumeration of Actions , 1996 .

[74]  Gabriel A Radvansky,et al.  Event perception. , 2011, Wiley interdisciplinary reviews. Cognitive science.

[75]  Tilo Kircher,et al.  Neural correlates of narrative shifts during auditory story comprehension , 2009, NeuroImage.

[76]  Jeffrey M. Zacks,et al.  Event structure in perception and conception. , 2001, Psychological bulletin.

[77]  G. Miller,et al.  Language and Perception , 1976 .

[78]  Monica Bucciarelli,et al.  Kinematic mental simulations in abduction and deduction , 2013, Proceedings of the National Academy of Sciences.

[79]  John R. Anderson How Can the Human Mind Occur in the Physical Universe , 2007 .

[80]  Resolving Valid Multiple Model Inferences Activates a Left Hemisphere Network , 2006 .

[81]  Ron Koppelberger,et al.  Space and Time , 2021, Nature.

[82]  T. Connolly The rational imagination: how people create alternatives to reality (1st ed., 284 pp.). By R. M. J. Byrne. Cambridge, MA: MIT Press, 2005. ISBN 0‐262‐02584‐1 , 2008 .

[83]  Jean Piaget,et al.  The child's conception of time; , 1969 .

[84]  W. H. F. Barnes The Nature of Explanation , 1944, Nature.