Higher visual areas act like domain-general filters with strong selectivity and functional specialization

Neuroscientific studies rely heavily on a-priori hypotheses, which can bias results toward existing theories. Here, we use a hypothesis-neutral approach to study category selectivity in higher visual cortex. Using only stimulus images and their associated fMRI activity, we constrain randomly initialized neural networks to predict voxel activity. Despite no category-level supervision, the units in the trained networks act as detectors for semantic concepts like ‘faces’ or ‘words’, providing solid empirical support for categorical selectivity. Importantly, this selectivity is maintained when training the networks without images that contain the preferred category, strongly suggesting that selectivity is not domain-specific machinery, but sensitivity to generic patterns that characterize preferred categories. The ability of the models’ representations to transfer to perceptual tasks further reveals the functional role of their selective responses. Finally, our models show selectivity only for a limited number of categories, all previously identified, suggesting that the essential categories are already known. Teaser Models trained solely to predict fMRI activity from images reveal strong category selectivity in higher visual areas, even without exposure to these categories in training.