Survey on the Analysis and Modeling of Visual Kinship: A Decade in the Making

Kinship recognition is a challenging problem with many practical applications. With much progress and milestones having been reached after ten years, we are now able to survey the research and create new milestones. We review the public resources and data challenges that enabled and inspired many to hone-in on the views of automatic kinship recognition in the visual domain. The different tasks are described in technical terms and syntax consistent across the problem domain and the practical value of each discussed and measured. State-of-the-art methods for visual kinship recognition problems, whether to discriminate between or generate from, are examined. As part of such, we review systems proposed as part of a recent data challenge held in conjunction with the 2020 IEEE Conference on Automatic Face and Gesture Recognition. We establish a stronghold for the state of progress for the different problems in a consistent manner. This survey will serve as the central resource for the work of the next decade to build upon. For the tenth anniversary, the demo code is provided for the various kin-based tasks. Detecting relatives with visual recognition and classifying the relationship is an area with high potential for impact in research and practice.

[1]  Ming Shao,et al.  Families in Wild Multimedia: A Multimodal Database for Recognizing Kinship , 2020, IEEE Transactions on Multimedia.

[2]  J. Paul Robinson,et al.  What Will Your Child Look Like? DNA-Net: Age and Gender Aware Kin Face Synthesizer , 2019, ArXiv.

[3]  Chen Chen,et al.  Visual Explanation for Deep Metric Learning , 2019, IEEE Transactions on Image Processing.

[4]  Timnit Gebru,et al.  Datasheets for datasets , 2018, Commun. ACM.

[5]  Lei Zhang,et al.  AdvKin: Adversarial Convolutional Network for Kinship Verification , 2017, IEEE Transactions on Cybernetics.

[6]  Helen Nissenbaum,et al.  An Ethical Highlighter for People-Centric Dataset Creation , 2020, ArXiv.

[7]  Andrei Shadrikov,et al.  Achieving Better Kinship Recognition Through Better Baseline , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[8]  Gerhard Rigoll,et al.  A Multi-Task Comparator Framework for Kinship Verification , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[9]  Abdelmalik Taleb-Ahmed,et al.  Multi-view Deep Features for Robust Facial Kinship Verification , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[10]  Jun Yu,et al.  Retrieval of Family Members Using Siamese Neural Network , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[11]  Jun Yu,et al.  Deep Fusion Siamese Network for Automatic Kinship Verification , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[12]  Zhipeng Luo,et al.  Challenge report Recognizing Families In the Wild Data Challenge , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[13]  P. Dayananda,et al.  Weighted full binary tree-sliced binary pattern: An RGB-D image descriptor , 2020, Heliyon.

[14]  Jie Zhou,et al.  Graph-Based Kinship Reasoning Network , 2020, 2020 IEEE International Conference on Multimedia and Expo (ICME).

[15]  Shaodi You,et al.  Kinship Identification through Joint Learning Using Kinship Verification Ensemble , 2020, ECCV.

[16]  Chetan Kumar,et al.  Adversary for Social Good: Protecting Familial Privacy through Joint Adversarial Attacks , 2020, AAAI.

[17]  Jinhui Tang,et al.  Deep multi-person kinship matching and recognition for family photos , 2020, Pattern Recognit..

[18]  Mohamed Lashab,et al.  Kinship Verification Through Facial Images Using CNN-Based Features , 2020, Traitement du Signal.

[19]  Yong Zhang,et al.  Controllable Descendant Face Synthesis , 2020, ArXiv.

[20]  Joseph P. Robinson,et al.  Face Recognition: Too Bias, or Not Too Bias? , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Joseph P. Robinson,et al.  Recognizing Families In the Wild (RFIW): The 4th Edition , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[22]  Recognizing Families In the Wild: White Paper for the 4th Edition Data Challenge. , 2020, 2002.06303.

[23]  Bruno G. C. Lima,et al.  Verifying Kinship from RGB-D Face Data , 2020, ACIVS.

[24]  Xiaoqian Qin,et al.  A literature survey on kinship verification through facial images , 2020, Neurocomputing.

[25]  Abdelmalik Taleb-Ahmed,et al.  Tensor cross-view quadratic discriminant analysis for kinship verification in the wild , 2020, Neurocomputing.

[26]  S. Zafeiriou,et al.  Analysing Affective Behavior in the First ABAW 2020 Competition , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[27]  Fady S. Ghatas,et al.  GANKIN: generating Kin faces using disentangled GAN , 2020, SN Applied Sciences.

[28]  F. Furstenberg.  Kinship Reconsidered: Research on a Neglected Topic. , 2020, Journal of marriage and the family.

[29]  Jorge Gonçalves,et al.  Augmenting Automated Kinship Verification with Targeted Human Input , 2020, PACIS.

[30]  Yunpeng Xiao,et al.  Attentive Sequences Recurrent Network for Social Relation Recognition from Video , 2019, IEICE Trans. Inf. Syst..

[31]  G. Hemantha Kumar,et al.  A Spatial Density and Phase Angle Based Correlation for Multi-type Family Photo Identification , 2019, ACPR.

[32]  Yun Fu,et al.  Cross-Generation Kinship Verification with Sparse Discriminative Metric , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Xuelong Li,et al.  Feature selection with multi-view data: A survey , 2019, Inf. Fusion.

[34]  Jie Zhou,et al.  Multi-View Geometric Mean Metric Learning for Kinship Verification , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[35]  Guodong Guo,et al.  Learning deep compact similarity metric for kinship verification from face images , 2019, Inf. Fusion.

[36]  Tong Lu,et al.  A geometric and fractional entropy-based method for family photo classification , 2019, Expert Syst. Appl. X.

[37]  Moumita Mukherjee,et al.  Kinship verification using Compound Local Binary Pattern and Local Feature Discriminant Analysis , 2019, 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT).

[38]  Tomi H. Kinnunen,et al.  Audio-Visual Kinship Verification in the Wild , 2019, 2019 International Conference on Biometrics (ICB).

[39]  C.-C. Jay Kuo,et al.  Deep Kinship Verification VIA Appearance-Shape Joint Prediction and Adaptation-Based Approach , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[40]  Abhilash Nandy,et al.  Kinship Verification using Deep Siamese Convolutional Neural Network , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[41]  Abdelmalik Taleb-Ahmed,et al.  Kinship Verification based Deep and Tensor Features through Extreme Learning Machine , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[42]  Xavier Binefa,et al.  Heatmap-Guided Balanced Deep Convolution Networks for Family Classification in the Wild , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[43]  Dong Wang,et al.  Social relationships classification using social contextual features and SVDD-based metric learning , 2019, Appl. Soft Comput..

[44]  Le Yu,et al.  Exploiting effective facial patches for robust gender recognition , 2019, Tsinghua Science and Technology.

[45]  Qinghua Hu,et al.  Weighted Graph Embedding-Based Metric Learning for Kinship Verification , 2019, IEEE Transactions on Image Processing.

[46]  Sixue Gong,et al.  Video Face Recognition: Component-wise Feature Aggregation Network (C-FAN) , 2019, 2019 International Conference on Biometrics (ICB).

[47]  John R. Smith,et al.  Diversity in Faces , 2019, ArXiv.

[48]  Mei Wang,et al.  Racial Faces in the Wild: Reducing Racial Bias by Information Maximization Adaptation Network , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[49]  Inioluwa Deborah Raji,et al.  Model Cards for Model Reporting , 2018, FAT.

[50]  Kush R. Varshney,et al.  Increasing Trust in AI Services through Supplier's Declarations of Conformity , 2018, IBM J. Res. Dev..

[51]  Richa Singh,et al.  Supervised Mixed Norm Autoencoder for Kinship Verification in Unconstrained Videos , 2018, IEEE Transactions on Image Processing.

[52]  Min Xu,et al.  Adversarial Similarity Metric Learning for Kinship Verification , 2019, IEEE Access.

[53]  Ying Sun,et al.  Video-based Parent-Child Relationship Prediction , 2018, 2018 IEEE Visual Communications and Image Processing (VCIP).

[54]  Mei Wang,et al.  Racial Faces in-the-Wild: Reducing Racial Bias by Deep Unsupervised Domain Adaptation , 2018, ArXiv.

[55]  Ming Shao,et al.  Visual Kinship Recognition of Families in the Wild , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Ming Shao,et al.  To Recognize Families In the Wild: A Machine Vision Tutorial , 2018, ACM Multimedia.

[57]  Yosi Keller,et al.  SelfKin: Self Adjusted Deep Model For Kinship Verification , 2018, ArXiv.

[58]  Andrew Zisserman,et al.  From Same Photo: Cheating on Visual Kinship Challenges , 2018, ACCV.

[59]  Jiwen Lu,et al.  Local Large-Margin Multi-Metric Learning for Face and Kinship Verification , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[60]  Abdenour Hadid,et al.  Kinship Verification using Color Features and Extreme Learning Machine , 2018, 2018 IEEE 3rd International Conference on Signal and Image Processing (ICSIP).

[61]  Savas Özkan,et al.  Kinshipgan: Synthesizing of Kinship Faces from Family Photos by Regularizing a Deep Face Network , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[62]  Jorge Gonçalves,et al.  Kinship verification from facial images and videos: human versus machine , 2018, Machine Vision and Applications.

[63]  Yun Fu,et al.  Kinship Classification through Latent Adaptive Subspace , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[64]  Nasser M. Nasrabadi,et al.  A Deep Face Identification Network Enhanced by Facial Attributes Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[65]  Richa Singh,et al.  Deep Face-Representation Learning for Kinship Verification , 2018 .

[66]  Haibin Yan,et al.  Video-based kinship verification using distance metric learning , 2018, Pattern Recognit..

[67]  Maja Pantic,et al.  Modelling of Facial Aging and Kinship: A Survey , 2018, Image Vis. Comput..

[68]  Jian Cheng,et al.  Additive Margin Softmax for Face Verification , 2018, IEEE Signal Processing Letters.

[69]  Shahar Mahpod,et al.  Kinship verification using multiview hybrid distance learning , 2017, Comput. Vis. Image Underst..

[70]  Omkar M. Parkhi,et al.  VGGFace2: A Dataset for Recognising Faces across Pose and Age , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[71]  Ahmed S. Mahmoud,et al.  Impact of Separation on Refugee Families: Syrian Refugees in Jordan , 2018 .

[72]  The ethics of catching criminals using their family’s DNA , 2018, Nature.

[73]  Dong Wang,et al.  Heterogeneous Similarity Learning for More Practical Kinship Verification , 2017, Neural Processing Letters.

[74]  Richa Singh,et al.  Hierarchical Representation Learning for Kinship Verification , 2017, IEEE Transactions on Image Processing.

[75]  S. Shan,et al.  KinNet: Fine-to-Coarse Deep Metric Learning for Kinship Verification , 2017 .

[76]  Handong Zhao,et al.  Recognizing Families In the Wild (RFIW): Data Challenge Workshop in conjunction with ACM MM 2017 , 2017 .

[77]  Qingyan Duan,et al.  AdvNet: Adversarial Contrastive Residual Net for 1 Million Kinship Recognition , 2017 .

[78]  Songyang Lao,et al.  Using Deep Relational Features to Verify Kinship , 2017, CCCV.

[79]  Lei Zhang,et al.  From Face Recognition to Kinship Verification: An Adaptation Approach , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[80]  Haijun Liu,et al.  Status-aware projection metric learning for kinship verification , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[81]  Md. Zakirul Alam Bhuiyan,et al.  A Survey on Deep Learning in Big Data , 2017, 22017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC).

[82]  Anil K. Jain,et al.  IARPA Janus Benchmark-B Face Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[83]  Jiwen Lu,et al.  Discriminative Deep Metric Learning for Face and Kinship Verification , 2017, IEEE Transactions on Image Processing.

[84]  Yun Fu,et al.  Kinship Verification on Families in the Wild with Marginalized Denoising Metric Learning , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[85]  Hamdi Dibeklioglu,et al.  What Will Your Future Child Look Like? Modeling and Synthesis of Hereditary Patterns of Facial Dynamics , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[86]  Yang Song,et al.  Age Progression/Regression by Conditional Adversarial Autoencoder , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[87]  K. Zhang,et al.  Kinship Verification with an Optimal CNN , 2017 .

[88]  Abdenour Hadid,et al.  On the usefulness of color for kinship verification from face images , 2016, 2016 IEEE International Workshop on Information Forensics and Security (WIFS).

[89]  Abdenour Hadid,et al.  Comments on the “Kinship Face in the Wild” Data Sets , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[90]  Yan Yan,et al.  Sparse similarity metric learning for kinship verification , 2016, 2016 Visual Communications and Image Processing (VCIP).

[91]  M. Kayser,et al.  Predicting Human Appearance from DNA for Forensic Investigations , 2016 .

[92]  Xiaoyi Feng,et al.  Kinship verification from faces: Methods, databases and challenges , 2016, 2016 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC).

[93]  Yuxiao Hu,et al.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition , 2016, ECCV.

[94]  Haibin Yan,et al.  Kinship verification from facial images by scalable similarity fusion , 2016, Neurocomputing.

[95]  Xiaoyi Feng,et al.  Kinship verification from videos using spatio-temporal texture features and deep learning , 2016, 2016 International Conference on Biometrics (ICB).

[96]  Ming Shao,et al.  Families in the Wild (FIW): Large-Scale Kinship Image Database and Benchmarks , 2016, ACM Multimedia.

[97]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[98]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[99]  Jiwen Lu,et al.  Prototype-Based Discriminative Feature Learning for Kinship Verification , 2015, IEEE Transactions on Cybernetics.

[100]  Jiwen Lu,et al.  The FG 2015 Kinship Verification in the Wild Evaluation , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[101]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[102]  Songcan Chen,et al.  Tri-Subject Kinship Verification: Understanding the Core of A Family , 2015, IEEE Transactions on Multimedia.

[103]  Liang Wang,et al.  Kinship Verification with Deep Convolutional Neural Networks , 2015, BMVC.

[104]  L. Maloney,et al.  Allocentric kin recognition is not affected by facial inversion. , 2015, Journal of vision.

[105]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[106]  Jiwen Lu,et al.  Kinship verification in the wild: The first kinship verification competition , 2014, IEEE International Joint Conference on Biometrics.

[107]  Hamdi Dibeklioglu,et al.  Graph-Based Kinship Recognition , 2014, 2014 22nd International Conference on Pattern Recognition.

[108]  Jiwen Lu,et al.  Discriminative Multimetric Learning for Kinship Verification , 2014, IEEE Transactions on Information Forensics and Security.

[109]  Afshin Dehghan,et al.  Who Do I Look Like? Determining Parent-Offspring Resemblance via Gated Autoencoders , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[110]  Albert Ali Salah,et al.  Like Father, Like Son: Facial Expression Dynamics for Kinship Verification , 2013, 2013 IEEE International Conference on Computer Vision.

[111]  Tsuhan Chen,et al.  Kinship classification by modeling facial feature heredity , 2013, 2013 IEEE International Conference on Image Processing.

[112]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[113]  Albert Ali Salah,et al.  Are You Really Smiling at Me? Spontaneous versus Posed Enjoyment Smiles , 2012, ECCV.

[114]  Jiebo Luo,et al.  Understanding Kin Relationships in a Photo , 2012, IEEE Transactions on Multimedia.

[115]  Jiwen Lu,et al.  Neighborhood repulsed metric learning for kinship verification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[116]  Andrea Bottino,et al.  A New Problem in Face Image Analysis - Finding Kinship Clues for Siblings Pairs , 2012, ICPRAM.

[117]  Ioannis A. Kakadiaris,et al.  Twins 3D face recognition challenge , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[118]  Ming Shao,et al.  Kinship Verification through Transfer Learning , 2011, IJCAI.

[119]  Nicola De Bellis,et al.  The Publish or Perish Book , 2011, J. Assoc. Inf. Sci. Technol..

[120]  Ming Shao,et al.  Genealogical face recognition based on UB KinFace database , 2011, CVPR 2011 WORKSHOPS.

[121]  Li Bai,et al.  Cosine Similarity Metric Learning for Face Verification , 2010, ACCV.

[122]  Gwenaël Kaminski,et al.  Firstborns’ Disadvantage in Kinship Detection , 2010, Psychological science.

[123]  Tsuhan Chen,et al.  Towards computational models of kinship verification , 2010, 2010 IEEE International Conference on Image Processing.

[124]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[125]  Charlie D. Frowd,et al.  Predict Your Child: a System to Suggest the Facial Appearance of Children , 2008, J. Multim..

[126]  Farhad B Naini,et al.  Three-dimensional assessment of the relative contribution of genetics and environment to various facial parameters with the twin method. , 2004, American journal of orthodontics and dentofacial orthopedics : official publication of the American Association of Orthodontists, its constituent societies, and the American Board of Orthodontics.

[127]  Juan Enrique Ramos,et al.  Using TF-IDF to Determine Word Relevance in Document Queries , 2003 .

[128]  J. Aldous.  World Revolution and Family Patterns , 2000 .

[129]  Robert M. French,et al.  Do Babies Resemble Their Fathers More Than Their Mothers? A Failure to Replicate Christenfeld and Hill (1995) , 1999 .

[130]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[131]  Yann LeCun,et al.  Signature Verification Using A "Siamese" Time Delay Neural Network , 1993, Int. J. Pattern Recognit. Artif. Intell..

[132]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.