Collection flow

Computing optical flow between any pair of Internet face photos is challenging for most current state of the art flow estimation methods due to differences in illumination, pose, and geometry. We show that flow estimation can be dramatically improved by leveraging a large photo collection of the same (or similar) object. In particular, consider the case of photos of a celebrity from Google Image Search. Any two such photos may have different facial expression, lighting and face orientation. The key idea is that instead of computing flow directly between the input pair (I, J), we compute versions of the images (I', J') in which facial expressions and pose are normalized while lighting is preserved. This is achieved by iteratively projecting each photo onto an appearance subspace formed from the full photo collection. The desired flow is obtained through concatenation of flows (I → I') o (J' → J). Our approach can be used with any two-frame optical flow algorithm, and significantly boosts the performance of the algorithm by providing invariance to lighting and shape changes.

[1]  Mark Tygert,et al.  A Randomized Algorithm for Principal Component Analysis , 2008, SIAM J. Matrix Anal. Appl..

[2]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[3]  Shree K. Nayar,et al.  Modeling the space of camera response functions , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Shahriar Negahdaripour,et al.  Revised Definition of Optical Flow: Integration of Radiometric and Geometric Cues for Dynamic Scene Analysis , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Li Zhang,et al.  Shape and motion under varying illumination , 2003, IEEE International Conference on Computer Vision.

[6]  Luc Van Gool,et al.  Unsupervised face alignment by robust nonrigid mapping , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  A. Shashua.  Geometry and Photometry in 3D Visual Recognition , 1992 .

[8]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[9]  David J. Kriegman,et al.  What Is the Set of Images of an Object Under All Possible Illumination Conditions? , 1998, International Journal of Computer Vision.

[10]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[11]  Alex Pentland,et al.  Photometric motion , 1991, [1990] Proceedings Third International Conference on Computer Vision.

[12]  Joachim Weickert,et al.  Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods , 2005, International Journal of Computer Vision.

[13]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[14]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Dani Lischinski,et al.  Non-rigid dense correspondence with applications for image enhancement , 2011, ACM Trans. Graph..

[16]  Gregory D. Hager,et al.  Efficient Region Tracking With Parametric Models of Geometry and Illumination , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Stefano Soatto,et al.  Real-time feature tracking and outlier rejection with changes in illumination , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[19]  Ronen Basri,et al.  Dense shape reconstruction of a moving object under arbitrary, unknown lighting , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[20]  Ronen Basri,et al.  Accuracy of Spherical Harmonic Approximations for Images of Lambertian Objects under Far and Near Lighting , 2004, ECCV.

[21]  David J. Kriegman,et al.  What is the set of images of an object under all possible lighting conditions? , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Takeo Kanade,et al.  Three-dimensional scene flow , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[23]  Ira Kemelmacher-Shlizerman,et al.  Face reconstruction in the wild , 2011, 2011 International Conference on Computer Vision.

[24]  David J. Fleet,et al.  Computing optical flow with physical models of brightness variation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[25]  Russell A. Epstein,et al.  5/spl plusmn/2 eigenimages suffice: an empirical investigation of low-dimensional lighting models , 1995, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[26]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[27]  Steven M. Seitz,et al.  The dimensionality of scene appearance , 2009, 2009 IEEE 12th International Conference on Computer Vision.