Face Recognition Vendor Test (FRVT) Part 8: Summarizing Demographic Differentials

In 2019, NIST Interagency Report 8280 quantified and visualized demographic variations for many face recognition algorithms. The report also suggested various mitigations, one of which - the focus of this report - was to define summary inequity measures that developers can work to improve and which can guide algorithm selection. Since 2019, it has become apparent that false negative inequities are substantially due to poor photography of certain groups including under-exposure of dark-skinned individiuals, and that this can be addressed by using algorithms more tolerant of poor image quality or, better, by correcting the capture process with superior cameras, imaging environments and human-factors. At the same time, it is also clear that the much larger false positive variations, which occur even in high-quality photographs, must be mitigated by algorithm developers. To those ends, this report compiles and analyzes various demographic summary measures for how face recognition false positive and false negative error rates differ across age, sex, and race-based demographic groups. We exercise some of the proposed measures by tabulating them for many algorithms submitted to the one-to-one comparison track of the Face Recognition Vendor Test. Those results appear on a regularly updated public webpage. This document is open for public comment until 2022-09-14. Comments should be directed to frvt@nist.gov. The measures in this report, and potentially others, are being considered for inclusion in the ISO/IEC 19795-10 standard on measurement and reporting of demographic effects in biometric systems, in particular towards forming standardized measures of disparate impact. European algorithm Chinese one