Sample Complexity of Uniform Convergence for Multicalibration

There is a growing interest in societal concerns in machine learning systems, especially in fairness. Multicalibration gives a comprehensive methodology to address group fairness. In this work, we address the multicalibration error and decouple it from the prediction error. The importance of decoupling the fairness metric (multicalibration) and the accuracy (prediction error) is due to the inherent trade-off between the two, and the societal decision regarding the "right tradeoff" (as imposed many times by regulators). Our work gives sample complexity bounds for uniform convergence guarantees of multicalibration error, which implies that regardless of the accuracy, we can guarantee that the empirical and (true) multicalibration errors are close. We emphasize that our results: (1) are more general than previous bounds, as they apply to both agnostic and realizable settings, and do not rely on a specific type of algorithm (such as deferentially private), (2) improve over previous multicalibration sample complexity bounds and (3) implies uniform convergence guarantees for the classical calibration error.

[1]  Nathan Srebro,et al.  Learning Non-Discriminatory Predictors , 2017, COLT.

[2]  Yishay Mansour,et al.  From External to Internal Regret , 2005, J. Mach. Learn. Res..

[3]  Shai Ben-David,et al.  Understanding Machine Learning - From Theory to Algorithms , 2014 .

[4]  Sergiu Hart,et al.  Smooth calibration, leaky forecasts, finite recall, and Nash dynamics , 2018, Games Econ. Behav..

[5]  Philip M. Long,et al.  Characterizations of Learnability for Classes of {0, ..., n}-Valued Functions , 1995, J. Comput. Syst. Sci..

[6]  Christopher Jung,et al.  Online Learning with an Unknown Fairness Metric , 2018, NeurIPS.

[7]  Seth Neel,et al.  Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness , 2017, ICML.

[8]  Christina Ilvento,et al.  Metric Learning for Individual Fairness , 2019, FORC.

[9]  Alexandra Chouldechova,et al.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments , 2016, Big Data.

[10]  Avi Feller,et al.  Algorithmic Decision Making and the Cost of Fairness , 2017, KDD.

[11]  Yang Liu,et al.  Calibrated Fairness in Bandits , 2017, ArXiv.

[12]  Aaron Roth,et al.  Fairness in Learning: Classic and Contextual Bandits , 2016, NIPS.

[13]  Max Simchowitz,et al.  The Implicit Fairness Criterion of Unconstrained Learning , 2018, ICML.

[14]  Christopher Jung,et al.  Metric-Free Individual Fairness in Online Learning , 2020, NeurIPS.

[15]  Guy N. Rothblum,et al.  Probably Approximately Metric-Fair Learning , 2018, ICML.

[16]  Esther Rolf,et al.  Delayed Impact of Fair Machine Learning , 2018, ICML.

[17]  Jon M. Kleinberg,et al.  Inherent Trade-Offs in the Fair Determination of Risk Scores , 2016, ITCS.

[18]  Jon M. Kleinberg,et al.  On Fairness and Calibration , 2017, NIPS.

[19]  Guy N. Rothblum,et al.  Fairness Through Computationally-Bounded Awareness , 2018, NeurIPS.

[20]  T. Cleary TEST BIAS: PREDICTION OF GRADES OF NEGRO AND WHITE STUDENTS IN INTEGRATED COLLEGES , 1968 .

[21]  Maria-Florina Balcan,et al.  Envy-Free Classification , 2018, NeurIPS.

[22]  Shai Ben-David,et al.  Multiclass Learnability and the ERM principle , 2011, COLT.

[23]  James Y. Zou,et al.  Multiaccuracy: Black-Box Post-Processing for Fairness in Classification , 2018, AIES.

[24]  Guy N. Rothblum,et al.  Multicalibration: Calibration for the (Computationally-Identifiable) Masses , 2018, ICML.

[25]  Toniann Pitassi,et al.  Fairness through awareness , 2011, ITCS '12.