Face recognition under variable pose and illumination conditions using 3D facial appearance models

This paper proposes a method in which an appearance model is constructed from 3D facial shape data that can describe the image variation due to the arbitrariness of the pose and the variation of the illumination, after which recognition is performed by fitting the model to facial images when the illumination conditions and the precise pose are unknown. The appearance model in the proposed method is constructed as follows. The illumination basis for an image of an arbitrary pose is derived from the geodesic illumination basis describing the brightness on the 3D object surface depending on the illumination variations, and an image with the same illumination condition is reproduced as the target image for the unknown illumination conditions. By optimizing the pose to minimize the reproduction error, model fitting with high accuracy is realized even if the exact pose is unknown. By experiment, the number of illumination samples needed in the calculation of the geodesic illumination basis is evaluated and it is verified that the proposed appearance model can describe arbitrary illumination variations in images with various poses. The robustness of the proposed pose optimization method against initial pose estimate is also evaluated. The effectiveness of the proposed method is demonstrated by a recognition experiment using 14,000 facial images taken in various situations, including extreme illumination variations such as backlighting, for a wide range of poses from frontal to 45° upward and 60° sidewise. © 2007 Wiley Periodicals, Inc. Syst Comp Jpn, 38(2): 57– 70, 2007; Published online in Wiley InterScience (). DOI 10.1002sscj.20646