We show that a large and realistic face data set can be built fr om news photographs and their associated captions. Our automatically constructed face d ta set consists of 30,281 face images, obtained by applying a face finder to approximately half a mil lion captioned news images. The faces are labeled using image information from the photogra phs and word information extracted from the corresponding caption. This data set is more realis tic than usual face recognition data sets, because it contains faces captured “in the wild” under a wide range of positions, poses, facial expressions, and illuminations. After faces are extracted from the images, and names with context are extracted from the associated caption, our system u ses a clustering procedure to find the correspondence between faces and their associated names in th picture-caption pairs. The context in which a name appears in a caption provides powe rful cues as to whether it is depicted in the associated image. By incorporating simple n atural language techniques, we are able to improve our name assignment significantly. We use two mode ls of word context, a naive Bayes model and a maximum entropy model. Once our procedure is comp lete, we have an accurately labeled set of faces, an appearance model for each individua l depicted, and a natural language model that can produce accurate results on captions in isola tion. keywords: Names; Faces; News; Words; Pictures