Real-time face relighting via adaptive normal mapping

In this paper, we present a real-time face relighting method for enhancing widely used face photographs such as `selfies'. In contrast with previous relighting methods that estimate and replace the real lighting condition with the intended one, our technique focuses on the aesthetic aspects of relighting with real-time interaction. We estimate the head pose based on the facial features and apply adaptive face normal mapping by transforming and interpolating pre-defined face normal maps. The lighting effect is calculated using adaptively mapped normals and overlaid on the original photograph. The experimental results show that the proposed method generates aesthetic face photographs without a computational overhead, enabling real-time face relighting for mobile applications.

[1]  Andrew Gardner,et al.  A lighting reproduction approach to live-action compositing , 2002, SIGGRAPH.

[2]  Peter Comninos.  Mathematical and computer programming techniques for computer graphics , 2005 .

[3]  Hideo Saito,et al.  Arbitrarily Shaped Objects Relighting Using an RGB-D Camera , 2013, 2013 2nd IAPR Asian Conference on Pattern Recognition.

[4]  Sung Yong Shin,et al.  Scattered Data Interpolation with Multilevel B-Splines , 1997, IEEE Trans. Vis. Comput. Graph..

[5]  Yunjin Lee,et al.  Structure grid for directional stippling , 2011, Graph. Model..

[6]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[7]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[8]  Tatsuo Arai,et al.  Face relighting using discriminative 2D spherical spaces for face recognition , 2013, Machine Vision and Applications.

[9]  Hideo Saito,et al.  Illumination Estimation and Relighting using an RGB-D Camera , 2015, VISAPP.

[10]  Rama Chellappa,et al.  Illumination robust dictionary-based face recognition , 2011, 2011 18th IEEE International Conference on Image Processing.

[11]  J. Alex Stark,et al.  Adaptive image contrast enhancement using generalizations of histogram equalization , 2000, IEEE Trans. Image Process..

[12]  A. Shafie,et al.  Geometrical Substantiation of Phi , the Golden Ratio and the Baroque of Nature, Architecture, Design and Engineering , 2012 .

[13]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[14]  Gang Hua,et al.  Face Re-Lighting from a Single Image under Harsh Lighting Conditions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Jiansheng Chen,et al.  Face Image Relighting using Locally Constrained Global Optimization , 2010, ECCV.

[16]  Alexei A. Efros,et al.  Fast bilateral filtering for the display of high-dynamic-range images , 2002 .

[17]  Tatsuo Arai,et al.  2D Spherical Spaces for Face Relighting under Harsh Illumination , 2012 .

[18]  Geoffrey E. Hinton,et al.  Deep Lambertian Networks , 2012, ICML.

[19]  John D. Fernandez,et al.  Facial feature detection using Haar classifiers , 2006 .

[20]  Hujun Yin,et al.  Eigenlights: Recovering Illumination from Face Images , 2011, IDEAL.

[21]  Yunjin Lee,et al.  Art‐photographic detail enhancement , 2014, Comput. Graph. Forum.

[22]  Sylvain Paris,et al.  Deep Photo Style Transfer , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jiawen Chen,et al.  Real-time edge-aware image processing with the bilateral grid , 2007, ACM Trans. Graph..

[24]  Kostadin Dabov,et al.  BM3D Image Denoising with Shape-Adaptive Principal Component Analysis , 2009 .

[25]  Yizhou Yu,et al.  Example-based image color and tone style enhancement , 2011, ACM Trans. Graph..

[26]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[27]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[28]  Long Quan,et al.  Lightweight face relighting , 2003, 11th Pacific Conference onComputer Graphics and Applications, 2003. Proceedings..

[29]  Larry S. Davis,et al.  Generalized Deep Image to Image Regression , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Pat Hanrahan,et al.  Conveying shape and features with image-based relighting , 2003, IEEE Visualization, 2003. VIS 2003..

[31]  Chu-Song Chen,et al.  Lighting normalization with generic intrinsic illumination subspace for face recognition , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[32]  Jan Kautz,et al.  Local Laplacian filters: edge-aware image processing with a Laplacian pyramid , 2011, SIGGRAPH 2011.

[33]  Gee-Sern Hsu,et al.  Face recognition using sparse representation with illumination normalization and component features , 2013, 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.

[34]  Ioannis A. Kakadiaris,et al.  Bidirectional relighting for 3D-aided 2D face recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Anil K. Jain,et al.  Face Detection in Color Images , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  N K Bansode,et al.  Face Shape Classification Based on Region Similarity, Correlation and Fractal Dimensions , 2016 .

[37]  Claus B. Madsen,et al.  A Model-Based Approach to Image Relighting with a Potential for Real-Time Implementation , 2005, VVG.

[38]  Yizhou Yu,et al.  An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition , 2015, ACM Trans. Graph..

[39]  Gang Hua,et al.  Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Jiebo Luo,et al.  A Subspace Model-Based Approach to Face Relighting Under Unknown Lighting and Poses , 2008, IEEE Transactions on Image Processing.

[41]  James F. Blinn,et al.  Simulation of wrinkled surfaces , 1978, SIGGRAPH.