Classification-based video super-resolution using artificial neural networks

In this study, a classification-based video super-resolution method using artificial neural network (ANN) is proposed to enhance low-resolution (LR) to high-resolution (HR) frames. The proposed method consists of four main steps: classification, motion-trace volume collection, temporal adjustment, and ANN prediction. A classifier is designed based on the edge properties of a pixel in the LR frame to identify the spatial information. To exploit the spatio-temporal information, a motion-trace volume is collected using motion estimation, which can eliminate unfathomable object motion in the LR frames. In addition, temporal lateral process is employed for volume adjustment to reduce unnecessary temporal features. Finally, ANN is applied to each class to learn the complicated spatio-temporal relationship between LR and HR frames. Simulation results show that the proposed method successfully improves both peak signal-to-noise ratio and perceptual quality.

[1]  Xuelong Li,et al.  A multi-frame image super-resolution method , 2010, Signal Process..

[2]  A. Murat Tekalp,et al.  Super resolution recovery for multi-camera surveillance imaging , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[3]  Jin-Jang Leou,et al.  Video super-resolution reconstruction using a mobile search strategy and adaptive patch size , 2010, 2010 International Conference on Signal Processing and Multimedia Applications (SIGMAP).

[4]  George Wolberg,et al.  Digital image warping , 1990 .

[5]  Sang Uk Lee,et al.  Image vector quantizer based on a classification in the DCT domain , 1991, IEEE Trans. Commun..

[6]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[7]  Liangpei Zhang,et al.  A MAP Approach for Joint Motion Estimation, Segmentation, and Super Resolution , 2007, IEEE Transactions on Image Processing.

[8]  Truong Q. Nguyen,et al.  Image Superresolution Using Support Vector Regression , 2007, IEEE Transactions on Image Processing.

[9]  Pramod K. Varshney,et al.  Super-resolution land cover mapping using a Markov random field based approach , 2005 .

[10]  J. Stark.  A neural network to compute the Hutchinson metric in fractal image processing , 1991, IEEE Trans. Neural Networks.

[11]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[12]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[13]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[14]  José Carlos M. Bermudez,et al.  Informed Choice of the LMS Parameters in Super-Resolution Video Reconstruction Applications , 2008, IEEE Transactions on Signal Processing.

[15]  G. Elisabeta Marai,et al.  Super-resolution registration using tissue-classified distance fields , 2006, IEEE Transactions on Medical Imaging.

[16]  Liangpei Zhang,et al.  A super-resolution reconstruction algorithm for surveillance images , 2010, Signal Process..

[17]  Liangpei Zhang,et al.  A super-resolution reconstruction algorithm for hyperspectral images , 2012, Signal Process..

[18]  Michael Elad,et al.  Generalizing the Nonlocal-Means to Super-Resolution Reconstruction , 2009, IEEE Transactions on Image Processing.

[19]  Yaron Caspi,et al.  Under the supervision of , 2003 .

[20]  Michael Margaliot,et al.  Are artificial neural networks white boxes? , 2005, IEEE Transactions on Neural Networks.

[21]  N. Sudha,et al.  A Self-Configurable Systolic Architecture for Face Recognition System Based on Principal Component Neural Network , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  Feng Li,et al.  Super Resolution for Remote Sensing Images Based on a Universal Hidden Markov Tree Model , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Kenneth E. Barner,et al.  A Computationally Efficient Super-Resolution Algorithm for Video Processing Using Partition Filters , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[25]  Liang Tang,et al.  Spatially Adaptive Block-Based Super-Resolution , 2012, IEEE Transactions on Image Processing.

[26]  Liangpei Zhang,et al.  Adaptive Multiple-Frame Image Super-Resolution Based on U-Curve , 2010, IEEE Transactions on Image Processing.

[27]  Kin-Man Lam,et al.  Example-based image super-resolution with class-specific predictors , 2009, J. Vis. Commun. Image Represent..

[28]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[29]  Mads Nielsen,et al.  Video Super-Resolution Using Simultaneous Motion and Intensity Calculations , 2011, IEEE Transactions on Image Processing.

[30]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[31]  Nirmal K. Bose,et al.  Recursive reconstruction of high resolution image from noisy undersampled multiframes , 1990, IEEE Trans. Acoust. Speech Signal Process..

[32]  J Jiang,et al.  Medical image analysis with artificial neural networks , 2010, Comput. Medical Imaging Graph..

[33]  Michael Elad,et al.  Super-Resolution Without Explicit Subpixel Motion Estimation , 2009, IEEE Transactions on Image Processing.

[34]  Haim Azhari,et al.  Super-resolution in PET imaging , 2006, IEEE Transactions on Medical Imaging.

[35]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[36]  Michael Elad,et al.  On the origin of the bilateral filter and ways to improve it , 2002, IEEE Trans. Image Process..