Sparse representation of data

The amount of electronic data available today as well as its dimensionality and complexity increases rapidly in many scientific areas including biology, (bio-)chemistry, medicine, physics and its application fields like robotics, bioinformatics or multimedia technologies. Many of these data sets are very complex but have also a simple inherent structure which allows an appropriate sparse representation and modeling of such data with less or no information loss. Advanced methods are needed to extract these inherent but hidden information. Sparsity can be observed at different levels: sparse representation of data points using e.g. dimension- ality reduction for efficient data storage, sparse representation of full data sets using e.g. prototypes to achieve compact models for lifelong learning and sparse models of the underlying data structure using sparse encoding techniques. One main goal is to achieve a human-interpretable represen- tation of the essential information. Sparse representations account for the ubiquitous problem that humans have to deal with ever increasing and inherently unlimited information by means of limited resources such as limited time, memory, or perception abilities. Starting with the seminal paper of Olshausen&Field (40) researchers recognized that sparsity can be used as a fundamental principle to arrive at very efficient information pro- cessing models for huge and complex data such as observed e.g. in the visual cortex. Nowadays, sparse models include diverse methods such as relevance learning in prototype based representations, sparse coding neu- ral gas, factor analysis methods, latent semantic indexing, sparse Bayesian networks, relevance vector machines and other. This tutorial paper reviews recent developments in the field.