KNG: The K-Norm Gradient Mechanism

This paper presents a new mechanism for producing sanitized statistical summaries that achieve \emph{differential privacy}, called the \emph{K-Norm Gradient} Mechanism, or KNG. This new approach maintains the strong flexibility of the exponential mechanism, while achieving the powerful utility performance of objective perturbation. KNG starts with an inherent objective function (often an empirical risk), and promotes summaries that are close to minimizing the objective by weighting according to how far the gradient of the objective function is from zero. Working with the gradient instead of the original objective function allows for additional flexibility as one can penalize using different norms. We show that, unlike the exponential mechanism, the noise added by KNG is asymptotically negligible compared to the statistical error for many problems. In addition to theoretical guarantees on privacy and utility, we confirm the utility of KNG empirically in the settings of linear and quantile regression through simulations.

[1]  Or Sheffet,et al.  Differentially Private Ordinary Least Squares , 2015, ICML.

[2]  Adam D. Smith,et al.  Privacy-preserving statistical estimation with optimal convergence rates , 2011, STOC '11.

[3]  C. Dwork,et al.  Exposed! A Survey of Attacks on Private Data , 2017, Annual Review of Statistics and Its Application.

[4]  Anand D. Sarwate,et al.  A near-optimal algorithm for differentially-private principal components , 2012, J. Mach. Learn. Res..

[5]  Philip B. Stark.  Privacy, Big Data, and the Public Good: Frameworks for Engagement , 2016 .

[6]  Aleksandra Slavkovic,et al.  Structure and Sensitivity in Differential Privacy: Comparing K-Norm Mechanisms , 2018, Journal of the American Statistical Association.

[7]  Assaf Schuster,et al.  Data mining with differential privacy , 2010, KDD.

[8]  Matthew Reimherr,et al.  Benefits and Pitfalls of the Exponential Mechanism with Applications to Hilbert Spaces and Functional PCA , 2019, ICML.

[9]  Adam D. Smith,et al.  The structure of optimal private tests for simple hypotheses , 2018, STOC.

[10]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[11]  Daniel Kifer,et al.  Private Convex Empirical Risk Minimization and High-dimensional Regression , 2012, COLT 2012.

[12]  Anand D. Sarwate,et al.  Differentially Private Empirical Risk Minimization , 2009, J. Mach. Learn. Res..

[13]  Naomi S. Altman,et al.  Quantile regression , 2019, Nature Methods.

[14]  L. Wasserman,et al.  A Statistical Framework for Differential Privacy , 2008, 0811.2501.

[15]  Alexander J. Smola,et al.  Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo , 2015, ICML.

[16]  Bing-Rong Lin,et al.  An Axiomatic View of Statistical Privacy and Utility , 2012, J. Priv. Confidentiality.

[17]  Thomas Steinke,et al.  Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds , 2016, TCC.

[18]  Kunal Talwar,et al.  On the geometry of differential privacy , 2009, STOC '10.

[19]  Philip S. Yu,et al.  A General Survey of Privacy-Preserving Data Mining Models and Algorithms , 2008, Privacy-Preserving Data Mining.

[20]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[21]  Anand D. Sarwate,et al.  Stochastic gradient descent with differentially private updates , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[22]  Hiroshi Nakagawa,et al.  Differential Privacy without Sensitivity , 2016, NIPS.

[23]  Kamalika Chaudhuri,et al.  Privacy-preserving logistic regression , 2008, NIPS.

[24]  Aaron Roth,et al.  A learning theory approach to non-interactive database privacy , 2008, STOC.

[25]  Kunal Talwar,et al.  Mechanism Design via Differential Privacy , 2007, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[26]  Martin J. Wainwright,et al.  Local privacy and statistical minimax rates , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[27]  Cynthia Dwork,et al.  Differential privacy and robust statistics , 2009, STOC '09.

[28]  Vishesh Karwa,et al.  Finite Sample Differentially Private Confidence Intervals , 2017, ITCS.

[29]  Stanislav Minsker.  Geometric median and robust estimation in Banach spaces , 2013, 1308.1334.

[30]  Aleksandra B. Slavkovic,et al.  Differentially Private Uniformly Most Powerful Tests for Binomial Data , 2018, NeurIPS.

[31]  Alberto Tesi,et al.  On the Problem of Local Minima in Backpropagation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Aleksandar Nikolov,et al.  Lower Bounds for Differential Privacy from Gaussian Width , 2016, SoCG.

[33]  Yin Yang,et al.  Functional Mechanism: Regression Analysis under Differential Privacy , 2012, Proc. VLDB Endow..

[34]  Guy N. Rothblum,et al.  Concentrated Differential Privacy , 2016, ArXiv.