Optical Flow Constraints on Deformable Models with Applications to Face Tracking

Optical flow provides a constraint on the motion of a deformable model. We derive and solve a dynamic system incorporating flow as a hard constraint, producing a model-based least-squares optical flow solution. Our solution also ensures the constraint remains satisfied when combined with edge information, which helps combat tracking error accumulation. Constraint enforcement can be relaxed using a Kalman filter, which permits controlled constraint violations based on the noise present in the optical flow information, and enables optical flow and edge information to be combined more robustly and efficiently. We apply this framework to the estimation of face shape and motion using a 3D deformable face model. This model uses a small number of parameters to describe a rich variety of face shapes and facial expressions. We present experiments in extracting the shape and motion of a face from image sequences which validate the accuracy of the method. They also demonstrate that our treatment of optical flow as a hard constraint, as well as our use of a Kalman filter to reconcile these constraints with the uncertainty in the optical flow, are vital for improving the performance of our system.

[1]  William H. Press,et al.  Book-Review - Numerical Recipes in Pascal - the Art of Scientific Computing , 1989 .

[2]  Andrew Blake,et al.  Accurate, real-time, unadorned lip tracking , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[3]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[5]  Arthur E. Bryson,et al.  Applied Optimal Control , 1969 .

[6]  Alex Pentland,et al.  Recovery of Nonrigid Motion and Structure , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[8]  Audra E. Kosh,et al.  Linear Algebra and its Applications , 1992 .

[9]  H. F. Durrant-White Consistent integration and propagation of disparate sensor observations , 1987 .

[10]  Yacov Hel-Or,et al.  Constraint fusion for recognition and localization of articulated objects , 1996, International Journal of Computer Vision.

[11]  Ahmed A. Shabana,et al.  Dynamics of Multibody Systems , 2020 .

[12]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  David G. Lowe,et al.  Fitting Parameterized Three-Dimensional Models to Images , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  J. Salz,et al.  Algorithms for estimation of three-dimensional motion , 1985, AT&T Technical Journal.

[15]  Rama Chellappa,et al.  Estimation of Object Motion Parameters from Noisy Images , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Y. Bar-Shalom Tracking and data association , 1988 .

[17]  Arthur Gelb,et al.  Applied Optimal Estimation , 1974 .

[18]  Andrew Blake,et al.  Learning Dynamics of Complex Motions from Image Sequences , 1996, ECCV.

[19]  Dimitris N. Metaxas Physics-Based Deformable Models: Applications to Computer Vision, Graphics, and Medical Imaging , 1996 .

[20]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[21]  Pascal Fua,et al.  Imposing Hard Constraints on Deformable Models through Optimization in Orthogonal Subspaces , 1997, Comput. Vis. Image Underst..

[22]  Demetri Terzopoulos Physically-Based Fusion of Visual Data over Space, Time, and Scale , 1993 .

[23]  Rajeev Sharma,et al.  Reliable tracking of human arm dynamics by multiple cue integration and constraint fusion , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[24]  Hugh F. Durrant-Whyte,et al.  Consistent Integration and Propagation of Disparate Sensor Observations , 1986, Proceedings. 1986 IEEE International Conference on Robotics and Automation.

[25]  Alan L. Yuille,et al.  Feature extraction from faces using deformable templates , 2004, International Journal of Computer Vision.

[26]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Edward H. Adelson,et al.  Probability distributions of optical flow , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  Larry S. Davis,et al.  Computing spatio-temporal representations of human faces , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[29]  C. Roberts Practical Anthropometry , 1888, British medical journal.

[30]  Gilad Adiv,et al.  Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Olivier D. Faugeras,et al.  Building, Registrating, and Fusing Noisy Visual Maps , 1988, Int. J. Robotics Res..

[32]  Alex Pentland,et al.  Motion regularization for model-based head tracking , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[33]  Alex Pentland,et al.  Closed-form solutions for physically-based shape modeling and recognition , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Guanghua Zhang,et al.  Physical modeling and combination of range and intensity edge data , 1993 .

[35]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[36]  Dimitris N. Metaxas,et al.  Shape and Nonrigid Motion Estimation Through Physics-Based Synthesis , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Shahriar Negahdaripour,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 2004 .

[38]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[39]  D.J. Kriegman,et al.  Stereo vision and navigation in buildings for mobile robots , 1989, IEEE Trans. Robotics Autom..

[40]  Andrew Blake,et al.  Determining facial expressions in real time , 1995, Proceedings of IEEE International Conference on Computer Vision.

[41]  Kiyoharu Aizawa,et al.  Analysis and synthesis of facial image sequences in model-based image coding , 1994, IEEE Trans. Circuits Syst. Video Technol..

[42]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  L. Farkas Anthropometry of the head and face , 1994 .

[44]  Timothy F. Cootes,et al.  Automatic Interpretation and Coding of Face Images Using Flexible Models , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Pascal Fua,et al.  Object-centered surface reconstruction: Combining multi-image stereo and shading , 1995, International Journal of Computer Vision.

[47]  최우영,et al.  Stereo vision 및 응용 , 1994 .

[48]  Berthold K. P. Horn Robot vision , 1986, MIT electrical engineering and computer science series.

[49]  Berthold K. P. Horn,et al.  Direct methods for recovering motion , 1988, International Journal of Computer Vision.

[50]  Demetri Terzopoulos,et al.  Constraints on Deformable Models: Recovering 3D Shape and Nonrigid Motion , 1988, Artif. Intell..

[51]  L. Farkas,et al.  Anthropometric Facial Proportions in Medicine , 1986 .

[52]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.