Bipredictive video super-resolution using key-frames

Many scalable video coding systems use variable resolution frames to ena ble different decoding layers. Some of these systems also use frame down-sampling along with enhancement layers to reduce complexity. In order to do that, super-resolution methods associated with efficient interpolation processes may help to increase the q uality of low-resolution frames. This work presents a super-resolution technique based on key frames. The goal is to restor e the high-frequency information of down-sampled frames using high-resolution frames as references. The super-resolved f rames can be used in scalable video coders, in variable quality coders, or in the side information generation for distributed coders. Res ults indicate substantial improvements over previous schemes. I. I NTRODUCTION Super resolution is known as a process of obtaining a high res olution image from a set of low resolution observations [1]. A low resolution image is that in which there is a low density o f pixels, that is, a low number of pixels per unit area, result ing in poor details. In the opposite, high resolution images hav e high pixel density, offering better detail information. B y this way, a super resolution process receives a set of low resolution i mages that will be processed generating a high resolution im age. In practice, this low resolution images could be a sequence o f video frames or even different views of the same scene. A traditionally used super resolution method is the Bayesian one [2], in which all input images have the same resolution. A different approach was proposed in [3], in which a algorithm is used to extract examples of a database of images to increas e the resolution of a given image. In video compression, there is a trade off between quality an d rate required to represent an image. In this way, the better the quality desired, the greater the amount of bits to be spent. Usually, this rate-distortion relation is contro lled by a quantization parameter. Alternative video encoding syst em have been proposed to take care of this relation. For exam ple, a mixed resolution approach was proposed, in which just part o f frames are downsampled allowing scalable video coding [4] , or reducing complexity coding [5]. In these mixed resolutio n schemes [6],[7], two types of frames could be identified: fr ames that remain in high resolution format, called Key Frames (KF s); and the frames that have reduced resolution, the Non-Key Frames (NKFs). The KFs could be encoded like Intra frames (I) , Predicted (P) or even Bipredicted (B), and will be used as reference on NKFs encoding process. At the decoder side, high resolution frames are reconstruct ed, interleaved with low resolution ones, as shown in Figure 1. Therefore, itt’s desired to use neighbor KFs to enhance th e NKfs quality. This kind of solution could be applied in any encoding system in which a set of more degraded frames are int erleaved with a less degraded ones like, for example, mixed quality codecs [8]. Also, the proposed method is relevant in the side-information generation process for certain distr ibu ed video coding architectures [9]. KF NKF KF NKF NKF NKF NKF NKF KF Fig. 1. Exemple of a video sequence in a mixed spacial resolutio n codec. A sequence of key frames (KF), in high-resolution, is interleaved with non-key frames (NKF), in low-resolution. The solution proposed to enhance non-key frames quality by u sing high resolution ones is quite generic, covering differ ent video encoding scenarios. The main goal of this work is prese nts a efficient technique to enhance low-quality decoded fra mes by using high-quality information extracted from availabl e neighbor frames in full resolution.