Principal Component Analysis for the Whole Facial Image With Pigmentation Separation and Application to the Prediction of Facial Images at Various Ages

In this article, principal component analysis is applied to pigmentation distribution in the whole face to obtain feature values, and the relationship between the obtained feature vectors and age is estimated by multiple regression analysis to simulate the changes of facial images in women of ages 10 to 80. Since the human face receives more attention than other body parts, a change of a small quantity of the features in a face makes a large difference to its appearance. We can divide the features into two categories. One category is physical features such as skin condition and shape, and the other is physiological features, which are influenced by age and health. In the beauty industry, the synthesis of skin texture is based on these two kinds of feature values. Previous works have analyzed only small areas of skin texture. By morphing the shapes of facial images to that of an average face and extending the analyzed area to the whole face, the authors’ method can analyze pigmentation distributions in the whole face and simulate the appearance of a face as a function of changing the person’s age. c © 2014 Society for Imaging Science and Technology. [DOI: 10.2352/J.ImagingSci.Technol.2014.58.2.020503] INTRODUCTION Human faces receive a lot of attention in comparison with other body parts. We can obtain a great deal of information from their appearance, such as individual features, race, age, emotion, gender, and health condition. A change of a small quantity of features makes a large difference to the appearance of a face. Recently, applications that change the appearance of faces have been put into practical use and continue to be developed with the advance of technology in various fields. Women especially have a strong interest in the Received Aug. 28, 2013; accepted for publication July 9, 2014; published online Aug. 8, 2014. Associate Editor: Yeong-Ho Ha. 1062-3701/2014/58(2)/020503/11/$25.00 appearance of their face or skin, and so applications that improve facial appearance are in high demand. However, conventional applications simply improve appearance, and these applications do not consider personal features. In the beauty industry, applications that change the facial appearance by using facial images are also required to predict facial images under various conditions. Computer-suggested skin analyses are often applied. Computers measure skin information such as moisture level or analyze the roughness of skin texture. Additionally, computer-suggested simulations that change the facial appearance by using facial images are used to simulatemakeup or predict the effect of basic skin care products. With these computer applications, users who wish to change their skin texture with makeup can obtain simulation results without trial-and-error or long-term use of a product. These applications are expected to synthesize the skin texture physically or simulate skin color based on two kinds of feature value. One kind is physical features, such as individual qualities and structures obtained from faces, and the other kind is physiological features, such as age and health condition. A cosmetic simulator is one example of the above applications that can change the facial appearance. The simulator applies digital makeup with skin information on moisture level, texture, etc. The simulator can advise customers when they are choosing cosmetics. This application can use numerous techniques, such as obtaining, analyzing and synthesizing skin texture or facial structure information, and estimating skin color changes. The facial appearance has been reconstructed or simulated by previousmethods inmany studies. For example, three-dimensional facial data or two-dimensional facial images are used to reconstruct facial appearance,1,2 or the changes of facial appearance are simulated formakeup,3,4 for various ages,5 and for race.6 Typical examples of simulation J. Imaging Sci. Technol. 020503-1 Mar.-Apr. 2014 Toyota et al.: Principal component analysis for the whole facial image with pigmentation separation and application to the prediction of facial ... methods are introduced in the following. Makino et al. developed a practical lighting reproduction technique to reproduce the appearance of a face under arbitrary lighting conditions.1,2 They reproduced the appearance of a face by combining image-based components, including captured live video images, as well as model-based components, including three-dimensional shapes, surface normals and the bidirectional reflectance distribution function (BRDF). Scherbaum et al. presented computer-suggested makeup.3 They obtained detailed feature values such as the threedimensional structure, a normal map, subsurface scattering, specular and diffuse reflectance, and glossiness by using facial photographs taken with different light sources. Guo et al. also proposed a digital makeup system, in which makeup information was extracted from facial images with makeup already applied, and the extracted information was added to another facial image without makeup.4 Appropriate results for personal features could be obtained in all of these studies, because they used a person’s own three-dimensional structure. In these studies, however, large systems are required to obtain information and it is difficult to put them into practical use. On the other hand, Lantis et al. proposed a framework that can be used for the simulation of aging effects on new face images to predict how an individualmight look in the future or might have looked in the past.5 In their method, the facial structure is changed by applying principal component analysis (PCA) and a genetic algorithm to landmarks obtained from monochrome images. Chalothorn et al. extracted racial differences between Japanese and Thai people.6 They applied PCA to skin texture and structure for the classification of race. PCA is used extensively as a method to obtain feature values comparatively easily. In this article, PCA is applied for pigmentation distributions in whole facial images to obtain feature values. In a general case, RGB values are used for the processing of facial images into skin texture information. However, RGB values are dependent on changes in the light source or characteristics of the camera, and they do not consider the skin structure and properties. Skin color mainly consists of melanin and hemoglobin pigmentations. Tsumura et al. proposed a method to extract melanin pigmentation and hemoglobin pigmentation from a single skin color image by independent component analysis.7,8 By using independent component analysis, melanin and hemoglobin colors can be obtained without the effects of changed light sources or the characteristics of a camera. Okaguchi et al. obtained hierarchical pigmentation distributions from image pyramid analysis and set image histograms as feature values.9 Furthermore, they analyzed the principal components of skin unevenness by applying PCA to feature values in the histograms and simulated the skin texture, which has arbitrary physiological features based on multiple regression analysis between the physiological features and the feature values in the histograms. This research showed that skin texture having uneven pigmentation can be synthesized physically to achieve an appropriate Figure 1. Overview of the imaging system. skin appearance. However, processing by this method is restricted to small skin areas, not the whole face. In this article, we apply PCA to whole facial images by extending the skin areas to analyze the pigmentation unevenness in the whole face. Additionally, we use multiple regression analysis to simulate the facial texture, which has arbitrary physiological features based on the relationship between the obtained principal components and feature vectors. In the next section we describe our approach. First, we present the construction of a facial image database, and morph the facial images to an average face. Then we extract the melanin and hemoglobin pigmentations from a single skin color image by independent component analysis.7 Next, we describe the method to apply PCA to pigmentation distributions and the method to analyze the principal components of uneven pigmentation. Finally, we simulate the appearance of a face of arbitrary age after estimating the relationship between the obtained principal components and feature values by multiple regression analysis. In the third section, we discuss our results. In the fourth section, we present the conclusions of our research. PROPOSEDMETHOD This section shows the method to obtain feature values of pigmentation unevenness in the whole face and the method to simulate facial appearance with arbitrary physiological features. The overview of the process is as follows. Step 1. Construction of a facial image database. Step 2. Morphing of facial images to obtain an average face. Step 3. Extraction of melanin and hemoglobin pigmentations by independent component analysis. Step 4. Analysis of the principal components of uneven pigmentation by PCA. Step 5. Estimation of the appearance of a face by multiple regression analysis and synthesis of facial images. In the following, we describe the details of the above processes. Construction of a Facial Image Database We took photographs of women who ranged in age from 10 to 80 and constructed a database. The number of subjects was 202. Figure 1 shows an overview of the imaging system. In the imaging system, ambient light sources were blocked by blackout curtains. The four fluorescent lights were set so that they surrounded the camera as the light source. J. Imaging Sci. Technol. 020503-2 Mar.-Apr. 2014 Toyota et al.: Principal component analysis for the whole facial image with pigmentation separation and application to the prediction of facial ... Figure 2. Sample of a captured facial image in the database. Figure 3. Distribution of ages in the database. We took images using a digital camera (Nikon D3X) and used a c