Generalized Face Super-Resolution

Existing learning-based face super-resolution (hallucination) techniques generate high-resolution images of a single facial modality (i.e., at a fixed expression, pose and illumination) given one or set of low-resolution face images as probe. Here, we present a generalized approach based on a hierarchical tensor (multilinear) space representation for hallucinating high-resolution face images across multiple modalities, achieving generalization to variations in expression and pose. In particular, we formulate a unified tensor which can be reduced to two parts: a global image-based tensor for modeling the mappings among different facial modalities, and a local patch-based multiresolution tensor for incorporating high-resolution image details. For realistic hallucination of unregistered low-resolution faces contained in raw images, we develop an automatic face alignment algorithm capable of pixel-wise alignment by iteratively warping the probing face to its projection in the space of training face images. Our experiments show not only performance superiority over existing benchmark face super-resolution techniques on single modal face hallucination, but also novelty of our approach in coping with multimodal hallucination and its robustness in automatic alignment under practical imaging conditions.

[1]  Tamara G. Kolda,et al.  Orthogonal Tensor Decompositions , 2000, SIAM J. Matrix Anal. Appl..

[2]  Frédéric Champagnat,et al.  An Improved Observation Model for Super-Resolution Under Affine Motion , 2006, IEEE Transactions on Image Processing.

[3]  Demetri Terzopoulos,et al.  Multilinear image analysis for facial recognition , 2002, Object recognition supported by user interaction for service robots.

[4]  Xiaogang Wang,et al.  Hallucinating face by eigentransformation , 2005, IEEE Trans. Syst. Man Cybern. Part C.

[5]  Michael Elad,et al.  Multiframe demosaicing and super-resolution of color images , 2006, IEEE Transactions on Image Processing.

[6]  Yücel Altunbasak,et al.  Eigenface-domain super-resolution for face recognition , 2003, IEEE Trans. Image Process..

[7]  Demetri Terzopoulos,et al.  Multilinear Analysis of Image Ensembles: TensorFaces , 2002, ECCV.

[8]  Michael Elad,et al.  Advances and challenges in super‐resolution , 2004, Int. J. Imaging Syst. Technol..

[9]  Michael Elad,et al.  Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images , 1997, IEEE Trans. Image Process..

[10]  Yuxiao Hu,et al.  Face recognition using Laplacianfaces , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[12]  J. Besag On the Statistical Analysis of Dirty Pictures , 1986 .

[13]  D. Yeung,et al.  Super-resolution through neighbor embedding , 2004, CVPR 2004.

[14]  Takeo Kanade,et al.  Limits on super-resolution and how to break them , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[15]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[17]  Harry Shum,et al.  A two-step approach to hallucinating faces: global parametric model and local nonparametric model , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[18]  Nanning Zheng,et al.  Image hallucination with primal sketch priors , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[19]  Takeo Kanade,et al.  Hallucinating faces , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[20]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Lei Zhang,et al.  Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Takeo Kanade,et al.  High-zoom video hallucination by exploiting spatio-temporal regularities , 2004, CVPR 2004.

[23]  Seong-Whan Lee,et al.  Resolution enhancement of facial image based on top-down learning , 2003, IWVS '03.

[24]  Robert L. Stevenson,et al.  Extraction of high-resolution frames from video sequences , 1996, IEEE Trans. Image Process..

[25]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[26]  Andrew Blake,et al.  Super-resolution Enhancement of Video , 2003, AISTATS.

[27]  Andrew Zisserman,et al.  Super-resolution from multiple views using learnt image models , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[28]  Lisimachos P. Kondi,et al.  An image super-resolution algorithm for different error levels per frame , 2006, IEEE Transactions on Image Processing.

[29]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[30]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[31]  Shaogang Gong,et al.  Multi-Resolution Patch Tensor for Facial Expression Hallucination , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[32]  P. Burt Fast filter transform for image processing , 1981 .

[33]  Russell C. Hardie,et al.  Joint MAP registration and high-resolution image estimation using a sequence of undersampled images , 1997, IEEE Trans. Image Process..

[34]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[35]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[36]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[37]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  P. J. Burt,et al.  Fast Filter Transforms for Image Processing , 1981 .

[39]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, SIGGRAPH 2005.

[40]  Narendra Ahuja,et al.  Facial expression decomposition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[41]  Shaogang Gong,et al.  Multi-modal tensor face for simultaneous super-resolution and recognition , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[42]  Anil K. Jain,et al.  Matching 2.5D face scans to 3D models , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Dahua Lin,et al.  Hallucinating faces: TensorPatch super-resolution and coupled residue compensation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[44]  Peyman Milanfar,et al.  Statistical performance analysis of super-resolution , 2006, IEEE Transactions on Image Processing.

[45]  Harry Shum,et al.  Face Hallucination: Theory and Practice , 2007, International Journal of Computer Vision.