Disparate Vulnerability to Membership Inference Attacks

Abstract A membership inference attack (MIA) against a machine-learning model enables an attacker to determine whether a given data record was part of the model’s training data or not. In this paper, we provide an in-depth study of the phenomenon of disparate vulnerability against MIAs: unequal success rate of MIAs against different population subgroups. We first establish necessary and sufficient conditions for MIAs to be prevented, both on average and for population subgroups, using a notion of distributional generalization. Second, we derive connections of disparate vulnerability to algorithmic fairness and to differential privacy. We show that fairness can only prevent disparate vulnerability against limited classes of adversaries. Differential privacy bounds disparate vulnerability but can significantly reduce the accuracy of the model. We show that estimating disparate vulnerability by naïvely applying existing attacks can lead to overestimation. We then establish which attacks are suitable for estimating disparate vulnerability, and provide a statistical framework for doing so reliably. We conduct experiments on synthetic and real-world data finding significant evidence of disparate vulnerability in realistic settings.

[1]  Reza Shokri,et al.  Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks , 2018, ArXiv.

[2]  Carl A. Gunter,et al.  A Pragmatic Approach to Membership Inferences on Machine Learning Models , 2020, 2020 IEEE European Symposium on Security and Privacy (EuroS&P).

[3]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[4]  Alexandra Chouldechova,et al.  The Frontiers of Fairness in Machine Learning , 2018, ArXiv.

[5]  Toniann Pitassi,et al.  Fairness through awareness , 2011, ITCS '12.

[6]  Alexandra Chouldechova,et al.  Does mitigating ML's impact disparity require treatment disparity? , 2017, NeurIPS.

[7]  G. Greenleaf,et al.  2020 Ends a Decade of 62 New Data Privacy Laws , 2020 .

[8]  Matt Fredrikson,et al.  Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference , 2019, USENIX Security Symposium.

[9]  Catuscia Palamidessi,et al.  F-BLEAU: Fast Black-Box Leakage Estimation , 2019, 2019 IEEE Symposium on Security and Privacy (SP).

[10]  Andrew D. Selbst,et al.  Big Data's Disparate Impact , 2016 .

[11]  Z. Obermeyer,et al.  Predicting the Future - Big Data, Machine Learning, and Clinical Medicine. , 2016, The New England journal of medicine.

[12]  Mohamed Ali Kaafar,et al.  Modelling and Quantifying Membership Information Leakage in Machine Learning , 2020, ArXiv.

[13]  Bruno Ribeiro,et al.  Membership Inference Attacks and Defenses in Classification Models , 2020, CODASPY.

[14]  Reza Shokri,et al.  On the Privacy Risks of Model Explanations , 2019, AIES.

[15]  Ian Goldberg,et al.  Differentially Private Learning Does Not Bound Membership Inference , 2020, ArXiv.

[16]  Lingxiao Wang,et al.  Revisiting Membership Inference Under Realistic Assumptions , 2020, Proc. Priv. Enhancing Technol..

[17]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[18]  Michael Veale,et al.  Algorithms that remember: model inversion attacks and data protection law , 2018, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[19]  Suresh Venkatasubramanian,et al.  On the (im)possibility of fairness , 2016, ArXiv.

[20]  Alexandra Chouldechova,et al.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments , 2016, Big Data.

[21]  M. Kearns,et al.  Fairness in Criminal Justice Risk Assessments: The State of the Art , 2017, Sociological Methods & Research.

[22]  Pol Mac Aonghusa,et al.  Diffprivlib: The IBM Differential Privacy Library , 2019, ArXiv.

[23]  Somesh Jha,et al.  Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting , 2017, 2018 IEEE 31st Computer Security Foundations Symposium (CSF).

[24]  Ron Kohavi,et al.  Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid , 1996, KDD.

[25]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[26]  A. Lo,et al.  Consumer Credit Risk Models Via Machine-Learning Algorithms , 2010 .

[27]  Vitaly Shmatikov,et al.  Differential Privacy Has Disparate Impact on Model Accuracy , 2019, NeurIPS.

[28]  Richard Honeck,et al.  Experimental Design and Analysis , 2006 .

[29]  Reza Shokri,et al.  On the Privacy Risks of Algorithmic Fairness , 2020, ArXiv.

[30]  Geoff Gordon,et al.  Inherent Tradeoffs in Learning Fair Representations , 2019, NeurIPS.

[31]  Liwei Song,et al.  Systematic Evaluation of Privacy Risks of Machine Learning Models , 2020, USENIX Security Symposium.

[32]  Hanna M. Wallach,et al.  Fairlearn: A toolkit for assessing and improving fairness in AI , 2020 .

[33]  Cordelia Schmid,et al.  White-box vs Black-box: Bayes Optimal Strategies for Membership Inference , 2019, ICML.

[34]  Mario Fritz,et al.  ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models , 2018, NDSS.

[35]  Ashwin Machanavajjhala,et al.  Fair decision making using privacy-protected data , 2019, FAT*.

[36]  Anand D. Sarwate,et al.  Differentially Private Empirical Risk Minimization , 2009, J. Mach. Learn. Res..

[37]  Michael D. Ekstrand,et al.  Privacy for All: Ensuring Fair and Equitable Privacy Protections , 2018, FAT.

[38]  Preetum Nakkiran,et al.  Distributional Generalization: A New Kind of Generalization , 2020, ArXiv.

[39]  Ronitt Rubinfeld,et al.  On the learnability of discrete distributions , 1994, STOC '94.

[40]  Gorjan Alagic,et al.  #p , 2019, Quantum information & computation.

[41]  Nathan Srebro,et al.  Equality of Opportunity in Supervised Learning , 2016, NIPS.

[42]  Carmela Troncoso,et al.  The Bayes Security Measure , 2020, ArXiv.