Five Principles for Crowd-Source Experiments in Face Recognition

The past few years have seen impressive gains inlong standing and difficult problems in face recognition. Thesegains have come about through the use of deep learning algorithmsthat consist of multi-layered neural networks. In part,the success of these algorithms is due to the easy availability ofextremely large datasets of faces that are annotated and labelledby humans. The reliance on crowd-sourced data for machinelearning and algorithm evaluation raises methodological issuesthat are not widely appreciated in computer vision. Several ofthese issues have come to light in recent work using crowdsourcing to benchmark human face identification on largedatabases that are used to test face recognition algorithms. Wedefine and discuss these issues using face recognition as a casestudy. We focus on: a.) the characteristics of the human participants;b.) the difference between aggregate and fused measuresof human accuracy; and c.) the lack of standard methods forcontrolling critical characteristics of the “imposter” distributionin large and variably diverse data sets. We will show thatestimates of human accuracy can vary widely depending on howthese factors combine in any given evaluation.We conclude withrecommendations on best practices in mitigating this variabilityand arriving at stable estimates of ground truth acquired bycrowd-sourcing.