Context-Aware Person Identification in Personal Photo Collections

Identifying the people in photos is an important need for users of photo management systems. We present MediAssist, one such system which facilitates browsing, searching and semi-automatic annotation of personal photos, using analysis of both image content and the context in which the photo is captured. This semi-automatic annotation includes annotation of the identity of people in photos. In this paper, we focus on such person annotation, and propose person identification techniques based on a combination of context and content. We propose language modelling and nearest neighbor approaches to context-based person identification, in addition to novel face color and image color content-based features (used alongside face recognition and body patch features). We conduct a comprehensive empirical study of these techniques using the real private photo collections of a number of users, and show that combining context- and content-based analysis improves performance over content or context alone.

[1]  Mor Naaman,et al.  Context data in geo-referenced digital photo collections , 2004, MULTIMEDIA '04.

[2]  Michael L. Creech,et al.  FotoFile: a consumer multimedia organization and retrieval system , 1999, CHI '99.

[3]  Benjamin B. Bederson,et al.  Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition , 2007, Interact. Comput..

[4]  Marian Stewart Bartlett,et al.  Face recognition by independent component analysis , 2002, IEEE Trans. Neural Networks.

[5]  Witold Pedrycz,et al.  Face recognition: A study in information fusion using fuzzy integral , 2005, Pattern Recognit. Lett..

[6]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[7]  LONGBIN CHEN,et al.  Face Annotation for Family Photo Album Management , 2003, Int. J. Image Graph..

[8]  Thijs Westerveld,et al.  CWI at the TREC 2002 Video Track , 2002, TREC.

[9]  A. Smeaton,et al.  IDENTIFYING PERSON RE-OCCURRENCES FOR PERSONAL PHOTO MANAGEMENT APPLICATIONS , 2006 .

[10]  W. Bruce Croft.  Combining Approaches to Information Retrieval , 2002 .

[11]  John Adcock,et al.  Leveraging face recognition technology to find and organize photos , 2004, MIR '04.

[12]  T. E. Doerfler,et al.  The behaviour of some significance tests under experimental randomization , 1969 .

[13]  Kentaro Toyama,et al.  Geographic location tags on digital images , 2003, ACM Multimedia.

[14]  B. S. Manjunath,et al.  Color and texture descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[15]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[16]  Tsuhan Chen,et al.  Using Group Prior to Identify People in Consumer Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Mingjing Li,et al.  Automated annotation of human faces in family albums , 2003, MULTIMEDIA '03.

[18]  Neil O'Hare.  Semi-automatic person-annotation in context-aware personal photo collections , 2007 .

[19]  Noel E. O'Connor,et al.  A hybrid technique for face detection in color images , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[20]  Ramesh C. Jain,et al.  Automatic Person Annotation of Family Photo Album , 2006, CIVR.

[21]  Fred Stentiford,et al.  Using context and similarity for face and location identification , 2006, Electronic Imaging.

[22]  Dragomir Anguelov,et al.  Contextual Identity Recognition in Personal Photo Albums , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[24]  Alan F. Smeaton,et al.  MediAssist: Using Content-Based Analysis and Context to Manage Personal Photo Collections , 2006, CIVR.

[25]  Alan F. Smeaton,et al.  A Comparison of Score, Rank and Probability-Based Fusion Methods for Video Shot Retrieval , 2005, CIVR.

[26]  Mor Naaman,et al.  Leveraging context to resolve identity in photo albums , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[27]  Noel E. O'Connor,et al.  Detecting the presence of large buildings in natural images , 2005 .

[28]  Richard Szeliski,et al.  Finding People in Repeated Shots of the Same Scene , 2006, BMVC.

[29]  Kerry Rodden,et al.  How do people manage their digital photographs? , 2003, CHI '03.

[30]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[31]  Jong-Hak Lee,et al.  Analyses of multiple evidence combination , 1997, SIGIR '97.

[32]  Yuandong Tian,et al.  A Face Annotation Framework with Partial Clustering and Interactive Labeling , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Andreas Paepcke,et al.  Time as essence for photo browsing through personal digital libraries , 2002, JCDL '02.

[34]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[35]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[36]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Jiebo Luo,et al.  Beyond pixels: Exploiting camera metadata for photo classification , 2005, Pattern Recognit..