What Makes Kevin Spacey Look Like Kevin Spacey

We reconstruct a controllable model of a person from a large photo collection that captures his or her {\em persona}, i.e., physical appearance and behavior. The ability to operate on unstructured photo collections enables modeling a huge number of people, including celebrities and other well photographed people without requiring them to be scanned. Moreover, we show the ability to drive or {\em puppeteer} the captured person B using any other video of a different person A. In this scenario, B acts out the role of person A, but retains his/her own personality and character. Our system is based on a novel combination of 3D face reconstruction, tracking, alignment, and multi-texture modeling, applied to the puppeteering problem. We demonstrate convincing results on a large variety of celebrities derived from Internet imagery and video.

[1]  Andrew Jones,et al.  Digital Ira: creating a real-time photoreal digital actor , 2013, SIGGRAPH '13.

[2]  Luc Van Gool,et al.  Face/Off: live facial puppetry , 2009, SCA '09.

[3]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[5]  Alexei A. Efros,et al.  What makes Paris look like Paris? , 2015, Commun. ACM.

[6]  Leonid Sigal,et al.  Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines , 2011, NIPS.

[7]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[8]  G. Wahba.  Spline models for observational data , 1990 .

[9]  Ira Kemelmacher-Shlizerman,et al.  Being John Malkovich , 2010, ECCV.

[10]  Hanspeter Pfister,et al.  Face transfer with multilinear models , 2005, ACM Trans. Graph..

[11]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[12]  Ira Kemelmacher-Shlizerman,et al.  Illumination-Aware Age Progression , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Katherine B. Martin,et al.  Facial Action Coding System , 2015 .

[14]  Feng Xu,et al.  Controllable high-fidelity facial performance transfer , 2014, ACM Trans. Graph..

[15]  Wojciech Matusik,et al.  Video face replacement , 2011, ACM Trans. Graph..

[16]  Ira Kemelmacher-Shlizerman,et al.  Face reconstruction in the wild , 2011, 2011 International Conference on Computer Vision.

[17]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[18]  Ira Kemelmacher-Shlizerman,et al.  Total Moving Face Reconstruction , 2014, ECCV.

[19]  Ira Kemelmacher-Shlizerman,et al.  Exploring photobios , 2011, ACM Trans. Graph..

[20]  Patrick Pérez,et al.  Automatic Face Reenactment , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Frédéric H. Pighin,et al.  Expressive speech-driven facial animation , 2005, TOGS.

[22]  Ira Kemelmacher-Shlizerman,et al.  Collection flow , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Yaser Sheikh,et al.  Content retargeting using parameter-parallel facial layers , 2011, SCA '11.

[24]  Paul Debevec,et al.  The Light Stages and Their Applications to Photoreal Digital Actors , 2012, SIGGRAPH 2012.

[25]  Joachim Weickert,et al.  Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods , 2005, International Journal of Computer Vision.

[26]  Zicheng Liu,et al.  Expressive expression mapping with ratio images , 2001, SIGGRAPH.

[27]  Mark Pauly,et al.  Realtime performance-based facial animation , 2011, ACM Trans. Graph..

[28]  Fei Yang,et al.  Facial expression editing in video using a temporally-smooth factorization , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.