Deep learning approaches for real-time image super-resolution

Generating a high-resolution (HR) image from its corresponding low-resolution (LR) input is referred to image super-resolution (SR). Generally, HR images contain higher pixel densities and more details in comparison with LR images. Image SR has already shown significant performance in many applications, such as video surveillance, remote sensing, face recognition, and medical images. Benefiting from its broad application prospects, SR has attracted enormous interests and is one of the most interesting and active research topics in image processing and computer vision. Early research in SR mainly focused on the frequency domain. The LR image is transformed into the frequency domain by the Fourier transform or wavelet transform for SR reconstruction. The SR algorithms based on the frequency domain are intuitive and straightforward, but do not consider the degradation process and prior information of an image. Therefore, the reconstruction is not ideal in a complex environment. To address the drawbacks of the frequency domain-based methods, spatial domain-based SR methods have gradually become the focus of mainstream research. Current spatial domain-based methods are mainly divided into two categories: reconstruction-based and learning-based. Reconstruction-based methods are always combined with one or more well-designed priors to estimate the details missed in the reconstruction process. These methods can obtain good results in preserving edges on the premise that a rational prior has been imposed. Therefore, research has employed a variety of methods to establish reconstruction priors like the sharpening of edge details, regularization, or deconvolution. Learning-based methods have become a hot spot in SR research in recent years due to their ability to recover highfrequency information of images. Learning-based SR methods establish the mapping of LR image pixels to HR image pixels by learning the spatial structure relationship between HR and LR image and aggregate HR image pixels to reconstruct the HR images. Learning-based methods try to restore missing high-frequency image details by establishing an implicit relationship between LR patches and their corresponding HR patches via machine learning methods. These methods have attracted more and more attention due to their promising and visually desirable reconstruction results. It is a general idea to enhance SR quality by learning relationships from a large quantity of training data. However, applying over data might introduce spurious high frequencies, resulting in noise and blur details. Therefore, it is important to keep a balance between the size of training data and reconstruction visual effects. With the development of machine learning technologies, several learning models have been explored to solve the SR problem. Learning-based methods can be divided into five groups based on differences in their core ideas: neighbor embedding methods, sparse coding methods, self-exemplar methods, locally linear regression methods, and deep learning methods. Recently, due to remarkable advances in deep learning, deep neural networks for SR have shown promising performance in several applications. This special issue collects eight papers reporting the recent developments of deep learning in image SR. The paper entitled ‘‘Perceptual image quality using dual generative adversarial network’’ develops a variety of generative adversarial networks for image SR that contains two generators and two discriminators. The generators learn from the mixture of real and generated images distributions. This methodology is trained with the feature & Pourya Shamsolmoali pshams@sjtu.edu.cn