Social relationships classification using social contextual features and SVDD-based metric learning

Abstract Family relationship is an important concern in image-based social relationships recognition, and there are very limited attempts to tackle diverse social relationships in the literature. In this paper, we propose the problem of social relationships classification in which we aim to model three types of social relationships( e.g., family, colleagues and friends) in the images. To this end, we introduce two types of social contextual features to capture detailed information( e.g., geometry or appearance) in images. Moreover, we present a new Support Vector Data Description-based metric learning( SML) method for social relationships classification. Motivated by the fact that the images are unavoidably degraded by noise due to some variation factors such as illumination and pose, we aim to learn a robust distance metric to suppress noise and model the spatial structure among multiple entities, such that more discriminative information can be exploited for classification. We also extend our method to multiview version-MSML, which helps to exploit multiple features to improve the social relationships classification performance. Extensive experiments on our newly released social relationships database demonstrate the feasibility and effectiveness of our proposed methods.

[1]  Ming Shao,et al.  Families in the Wild (FIW): Large-Scale Kinship Image Database and Benchmarks , 2016, ACM Multimedia.

[2]  Qingming Huang,et al.  Friend recommendation according to appearances on photos , 2009, MM '09.

[3]  Amir-Masoud Eftekhari-Moghadam,et al.  Combination of classification and regression in decision tree for multi-labeling image annotation and retrieval , 2013, Appl. Soft Comput..

[4]  Jiebo Luo,et al.  Discovery of social relationships in consumer photo collections using Markov Logic , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[5]  Xiaolong Wang,et al.  Leveraging multiple cues for recognizing family photos , 2017, Image Vis. Comput..

[6]  Wei-Yun Yau,et al.  Family verification based on similarity of individual family member’s facial segments , 2013, Machine Vision and Applications.

[7]  Xiaoyang Tan,et al.  Tri-subjects kinship verification: Understanding the core of a family , 2015, 2015 14th IAPR International Conference on Machine Vision Applications (MVA).

[8]  Gang Wang,et al.  Seeing People in Social Context: Recognizing People and Social Relationships , 2010, ECCV.

[9]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[10]  Frédéric Jurie,et al.  PCCA: A new approach for distance learning from sparse pairwise constraints , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Haibin Yan,et al.  Kinship verification using neighborhood repulsed correlation metric learning , 2017, Image Vis. Comput..

[12]  Xiaoyang Tan,et al.  Robust Distance Metric Learning in the Presence of Label Noise , 2014, AAAI.

[13]  Xiaoyang Tan,et al.  Bayesian Neighborhood Component Analysis , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[15]  Jiwen Lu,et al.  Neighborhood repulsed metric learning for kinship verification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Li Bai,et al.  Cosine Similarity Metric Learning for Face Verification , 2010, ACCV.

[17]  Jian Yang,et al.  Feature fusion: parallel strategy vs. serial strategy , 2003, Pattern Recognit..

[18]  Sanjeev Kumar,et al.  Finding a Needle in Haystack: Facebook's Photo Storage , 2010, OSDI.

[19]  Xiaolong Wang,et al.  Leveraging geometry and appearance cues for recognizing family photos , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[20]  Jiwen Lu,et al.  Sharable and Individual Multi-View Metric Learning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Haibin Yan,et al.  Kinship verification from facial images by scalable similarity fusion , 2016, Neurocomputing.

[22]  Si-Yu Xia,et al.  A genetics-motivated unsupervised model for tri-subject kinship verification , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[23]  Geoffrey E. Hinton,et al.  Neighbourhood Components Analysis , 2004, NIPS.

[24]  Jiwen Lu,et al.  Local Large-Margin Multi-Metric Learning for Face and Kinship Verification , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[25]  Xiaolong Wang,et al.  Kinship Measurement on Salient Facial Features , 2012, IEEE Transactions on Instrumentation and Measurement.

[26]  Inderjit S. Dhillon,et al.  Information-theoretic metric learning , 2006, ICML '07.

[27]  Yongdong Zhang,et al.  Multiview Spectral Embedding , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Tsuhan Chen,et al.  Kinship classification by modeling facial feature heredity , 2013, 2013 IEEE International Conference on Image Processing.

[29]  Trevor Darrell,et al.  Autotagging Facebook: Social network context improves photo annotation , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[30]  Hong-Yuan Mark Liao,et al.  Discovering informative social subgraphs and predicting pairwise relationships from group photos , 2012, ACM Multimedia.

[31]  Michael I. Jordan,et al.  Multiple kernel learning, conic duality, and the SMO algorithm , 2004, ICML.

[32]  Tsuhan Chen,et al.  Understanding images of groups of people , 2009, CVPR.

[33]  Koby Crammer,et al.  Online Passive-Aggressive Algorithms , 2003, J. Mach. Learn. Res..

[34]  Ming Shao,et al.  Toward kinship verification using visual attributes , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).