Animated Heads: From 3D Motion Fields to Action Descriptions

We demonstrate a method to compute three-dimensional (3D) motion fields on a face. Twelve synchronized and calibrated cameras are po­ sitioned around a talking person, and observe its head in motion. We represent the head as a deformable mesh, which is fitted in aglobai optimization step to silhouette-contour and multi-camera stereo data derived from all images. The non-rigid displacement of the mesh from frame to frame, the 3D motion field, is determined from the spatio­ temporal derivatives in all the images. We integrate these cues over time, thus producing an animated representation of the talking head. Our ability to estimate 3D motion fields points to a new framework for the study of action. The 3D motion fields can serve as an intermediate representation, which can be analyzed using geometrical and statisti­ cal tools for the purpose of extracting representations of generic human actions.