On spatio-temporal sparse coding : Analysis and an algorithm

Sparse coding is a common approach to learning local features for object recognition. Recently, there has been an increasing interest in learning features from spatio-temporal, binocular, or other multi-observation data, where the goal is to encode the relationship between images rather than the content of a single image. We discuss the role of multiplicative interactions and of squaring non-linearities in learning such relations. In particular, we show that training a sparse coding model whose filter responses are squared amounts to jointly diagonalizing a set of image transformations. Inference amounts to detecting rotations in the shared eigenspaces. Our analysis helps explain recent experimental results showing that Fourier features and circular Fourier features emerge when training complex cell models on translating or rotating images. And it suggests that it will be crucial to include either squaring or cross-products into deep learning architectures if we want to extend their applicability beyond simple tasks like recognizing objects in a single image.