Hierarchical structure is employed by humans during visual motion perception

Significance The structured organization of motion in visual scenes is highly informative for our everyday perception: We recognize people by the way they walk, track objects through occlusion, or predict hazardous situations from the traffic flow. It is, however, unclear how our minds tame the overwhelmingly complex stream of dynamic information received by the retina to form such stable percepts. We argue that an observer can exploit a “divide-and-conquer” strategy where complex motion relations are broken down into compositions of simpler motions. Evidence for hierarchical decomposition comes from multiple object tracking and prediction experiments in which humans are able to exploit motion structure knowledge to improve performance. Our results can guide neuroscience experiments on the neural representation of structure. In the real world, complex dynamic scenes often arise from the composition of simpler parts. The visual system exploits this structure by hierarchically decomposing dynamic scenes: When we see a person walking on a train or an animal running in a herd, we recognize the individual’s movement as nested within a reference frame that is, itself, moving. Despite its ubiquity, surprisingly little is understood about the computations underlying hierarchical motion perception. To address this gap, we developed a class of stimuli that grant tight control over statistical relations among object velocities in dynamic scenes. We first demonstrate that structured motion stimuli benefit human multiple object tracking performance. Computational analysis revealed that the performance gain is best explained by human participants making use of motion relations during tracking. A second experiment, using a motion prediction task, reinforced this conclusion and provided fine-grained information about how the visual system flexibly exploits motion structure.