Fisher Information Under Local Differential Privacy

We develop data processing inequalities that describe how Fisher information from statistical samples can scale with the privacy parameter <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> under local differential privacy constraints. These bounds are valid under general conditions on the distribution of the score of the statistical model, and they elucidate under which conditions the dependence on <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> is linear, quadratic, or exponential. We show how these inequalities imply order-optimal lower bounds for private estimation for both the Gaussian location model and discrete distribution estimation for all levels of privacy <inline-formula> <tex-math notation="LaTeX">$\varepsilon >0$ </tex-math></inline-formula>. We further apply these inequalities to sparse Bernoulli models and demonstrate privacy mechanisms and estimators with order-matching squared <inline-formula> <tex-math notation="LaTeX">$\ell ^{2}$ </tex-math></inline-formula> error.

[1]  Peter Kairouz,et al.  Discrete Distribution Estimation under Local Privacy , 2016, ICML.

[2]  Angelika Rohde,et al.  Geometrizing rates of convergence under local differential privacy constraints , 2020 .

[3]  Martin J. Wainwright,et al.  Local privacy and statistical minimax rates , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[4]  Feng Ruan,et al.  The Right Complexity Measure in Locally Private Estimation: It is not the Fisher Information , 2018, ArXiv.

[5]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[6]  Himanshu Tyagi,et al.  Inference Under Information Constraints I: Lower Bounds From Chi-Square Contraction , 2018, IEEE Transactions on Information Theory.

[7]  John Duchi,et al.  Lower Bounds for Locally Private Estimation via Communication Complexity , 2019, COLT.

[8]  Liam Paninski,et al.  A Coincidence-Based Test for Uniformity Given Very Sparsely Sampled Discrete Data , 2008, IEEE Transactions on Information Theory.

[9]  C. R. Rao,et al.  Information and the Accuracy Attainable in the Estimation of Statistical Parameters , 1992 .

[10]  Sofya Raskhodnikova,et al.  What Can We Learn Privately? , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[11]  Yanjun Han,et al.  Fisher Information for Distributed Estimation under a Blackboard Communication Protocol , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[12]  Alexandre B. Tsybakov,et al.  Introduction to Nonparametric Estimation , 2008, Springer series in statistics.

[13]  Yanjun Han,et al.  Lower Bounds for Learning Distributions under Communication Constraints via Fisher Information , 2019 .

[14]  Cynthia Dwork,et al.  Differential privacy and robust statistics , 2009, STOC '09.

[15]  Claudia Biermann,et al.  Mathematical Methods Of Statistics , 2016 .

[16]  Yanjun Han,et al.  A Geometric Characterization of Fisher Information from Quantized Samples with Applications to Distributed Statistical Estimation , 2018, 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[17]  Alexander Barg,et al.  Optimal Schemes for Discrete Distribution Estimation Under Locally Differential Privacy , 2017, IEEE Transactions on Information Theory.

[18]  S L Warner,et al.  Randomized response: a survey technique for eliminating evasive answer bias. , 1965, Journal of the American Statistical Association.

[19]  R. Gill,et al.  Applications of the van Trees inequality : a Bayesian Cramr-Rao bound , 1995 .

[20]  Raef Bassily,et al.  Linear Queries Estimation with Local Differential Privacy , 2018, AISTATS.

[21]  Cynthia Dwork,et al.  Differential Privacy , 2006, ICALP.

[22]  Himanshu Tyagi,et al.  Distributed Simulation and Distributed Inference , 2018, Electron. Colloquium Comput. Complex..

[23]  Angelika Rohde,et al.  Geometrizing rates of convergence under differential privacy constraints , 2018, 1805.01422.

[24]  J. Hájek.  Local asymptotic minimax and admissibility in estimation , 1972 .

[25]  Thomas M. Cover,et al.  Elements of information theory (2. ed.) , 2006 .

[26]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.