Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraints

Classifiers can be trained with data-dependent constraints to satisfy fairness goals, reduce churn, achieve a targeted false positive rate, or other policy goals. We study the generalization performance for such constrained optimization problems, in terms of how well the constraints are satisfied at evaluation time, given that they are satisfied at training time. To improve generalization performance, we frame the problem as a two-player game where one player optimizes the model parameters on a training dataset, and the other player enforces the constraints on an independent validation dataset. We build on recent work in two-player constrained optimization to show that if one uses this two-dataset approach, then constraint generalization can be significantly improved. As we illustrate experimentally, this approach works not only in theory, but also in practice.

[1]  Shang-Hua Teng,et al.  Electrical flows, laplacian systems, and faster approximation of maximum flow in undirected graphs , 2010, STOC '11.

[2]  Karthik Sridharan,et al.  Two-Player Games for Efficient Non-Convex Constrained Optimization , 2018, ALT.

[3]  Shai Ben-David,et al.  Empirical Risk Minimization under Fairness Constraints , 2018, NeurIPS.

[4]  Maya R. Gupta,et al.  Monotonic Calibrated Interpolated Look-Up Tables , 2015, J. Mach. Learn. Res..

[5]  John Langford,et al.  A Reductions Approach to Fair Classification , 2018, ICML.

[6]  Seth Neel,et al.  Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness , 2017, ICML.

[7]  Richard G. Baraniuk,et al.  Tuning Support Vector Machines for Minimax and Neyman-Pearson Classification , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Dan A. Biddle Adverse Impact and Test Validation: A Practitioner's Guide to Valid and Defensible Employment Testing , 2005 .

[9]  Nathan Srebro,et al.  Learning Non-Discriminatory Predictors , 2017, COLT.

[10]  Karthik Sridharan,et al.  Optimization, Learning, and Games with Predictable Sequences , 2013, NIPS.

[11]  Andrew Y. Ng,et al.  Preventing "Overfitting" of Cross-Validation Data , 1997, ICML.

[12]  Yoram Singer,et al.  Pegasos: primal estimated sub-gradient solver for SVM , 2011, Math. Program..

[13]  Robert S. Chen,et al.  Robust Optimization for Non-Convex Objectives , 2017, NIPS.

[14]  Maya R. Gupta,et al.  Satisfying Real-world Goals with Dataset Constraints , 2016, NIPS.

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Sanjeev Arora,et al.  The Multiplicative Weights Update Method: a Meta-Algorithm and Applications , 2012, Theory Comput..

[17]  Léon Bottou,et al.  Batch and online learning algorithms for nonconvex neyman-pearson classification , 2011, TIST.

[18]  Krishna P. Gummadi,et al.  Fairness Constraints: A Mechanism for Fair Classification , 2015, ArXiv.

[19]  Harikrishna Narasimhan,et al.  Learning with Complex Loss Functions and Constraints , 2018, AISTATS.

[20]  Geoffrey J. Gordon,et al.  No-regret learning in convex games , 2008, ICML '08.

[21]  Nathan Srebro,et al.  Equality of Opportunity in Supervised Learning , 2016, NIPS.

[22]  Robert D. Nowak,et al.  A Neyman-Pearson approach to statistical learning , 2005, IEEE Transactions on Information Theory.

[23]  Maya R. Gupta,et al.  Fast and Flexible Monotonic Functions with Ensembles of Lattices , 2016, NIPS.