Compressed Sensing : Basic results and self contained proofs

Compressed sensing is a linear dimensionality reduction technique which utilizes a prior assumption that the original vector is (approximately) sparse in some basis. In this note we summarize some of the known results and provide self contained, easy to follow, proofs. 1 Motivation Consider a vector x ∈ R that has at most s non-zero elements. That is, ‖x‖0 def = |{i : xi 6= 0}| ≤ s . Clearly, we can compress x by representing it using s (index,value) pairs. Furthermore, this compression is lossless – we can reconstruct x exactly from the s (index,value) pairs. Now, lets take one step forward and assume that x = Uα, where α is a sparse vector, ‖α‖0 ≤ s, and U is a fixed orthonormal matrix. That is, x has a sparse representation in another basis. It turns out that many natural vectors are (at least approximately) sparse in some representation. In fact, this assumption underlies many modern compression schemes. For example, the JPEG-2000 format for image compression relies on the fact that natural images are approximately sparse in a wavelet basis. Can we still compress x into roughly s numbers? Well, one simple way to do this is to multiply x by U , which yields the sparse vector α, and then represent α by its s (index,value) pairs. However, this requires to first ’sense’ x, to store it, and then to multiply it by U . This raises a very natural question: Why go to so much effort to acquire all the data when most of what we get will be thrown away? Can’t we just directly measure the part that won’t end up being thrown away? Compressed sensing is a technique that simultaneously acquire and compress the data. The key result is that a random linear transformation can compress x without loosing information. The number of measurements needed is order of s log(d). That is, we roughly acquire only the important information about the signal. As we will see later, the price we pay is a slower reconstruction phase. In some situations, it makes sense to save time in compression even at the price of a slower reconstruction. For example, a security camera should sense and compress a large amount of images while most of the time we do not need to decode the compressed data at all. Furthermore, in many practical applications, compression by a linear transformation is advantageous because it can be performed efficiently in hardware. For example, a team led by Baraniuk and Kelly have proposed a camera architecture that employs a digital micromirror array to perform optical calculations of a linear transformation of an image. In this case, obtaining each compressed measurement is as easy as obtaining a single raw measurement. Another important application of compressed sensing is medical imaging, in which requiring less measurements translates to less radiation for the patient.