Super-Resolution of Facial Images in Video with Expression Changes

Super-resolution (SR) of facial images from video suffers from facial expression changes. Most of the existing SR algorithms for facial images make an unrealistic assumption that the ¿perfect¿ registration has been done prior to the SR process. However, the registration is a challenging task for SR with expression changes. This paper proposes a new method for enhancing the resolution of low-resolution (LR) facial image by handling the facial image in a non-rigid manner. It consists of global tracking, local alignment for precise registration and SR algorithms. A B-spline based resolution aware incremental free form deformation (RAIFFD) model is used to recover a dense local non-rigid flow field. In this scheme, low-resolution image model is explicitly embedded in the optimization function formulation to simulate the formation of low resolution image. The results achieved by the proposed approach are significantly better as compared to the SR approaches applied on the whole face image without considering local deformations. The results are also compared with two state-of-the-art SR algorithms to show the effectiveness of the approach in super-resolving facial images with local expression changes.

[1]  Zhaohui Wu,et al.  Super-Resolution of 3D Face , 2006, ECCV.

[2]  Edward Courtney,et al.  2 = 4 M , 1993 .

[3]  Dahua Lin,et al.  Layered local prediction network with dynamic learning for face super-resolution , 2005, IEEE International Conference on Image Processing 2005.

[4]  Xiaogang Wang,et al.  Hallucinating face by eigentransformation , 2005, IEEE Trans. Syst. Man Cybern. Part C.

[5]  Nikos Paragios,et al.  Shape registration in implicit spaces using information theory and free form deformations , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Gregory D. Hager,et al.  Efficient Region Tracking With Parametric Models of Geometry and Illumination , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Takeo Kanade,et al.  Limits on super-resolution and how to break them , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[8]  Dahua Lin,et al.  Hallucinating faces: TensorPatch super-resolution and coupled residue compensation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[9]  Song-Chun Zhu,et al.  Learning in Gibbsian fields: how accurate and how fast can it be? , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[10]  Shmuel Peleg,et al.  Robust super-resolution , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[11]  Michal Irani,et al.  Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency , 1993, J. Vis. Commun. Image Represent..

[12]  Takeo Kanade,et al.  High-zoom video hallucination by exploiting spatio-temporal regularities , 2004, CVPR 2004.

[13]  Andrew Zisserman,et al.  Super-resolution from multiple views using learnt image models , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[14]  Michael Elad,et al.  Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images , 1997, IEEE Trans. Image Process..

[15]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[16]  Shaogang Gong,et al.  Hallucinating multiple occluded face images of different resolutions , 2006, Pattern Recognit. Lett..

[17]  Harry Shum,et al.  A two-step approach to hallucinating faces: global parametric model and local nonparametric model , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.