CheckDP: An Automated and Integrated Approach for Proving Differential Privacy or Finding Precise Counterexamples

We propose CheckDP, an automated and integrated approach for proving or disproving claims that a mechanism is differentially private. CheckDP can find counterexamples for mechanisms with subtle bugs for which prior counterexample generators have failed. Furthermore, it was able to automatically generate proofs for correct mechanisms for which no formal verification was reported before. CheckDP is built on static program analysis, allowing it to be more efficient and precise in catching infrequent events than sampling based counterexample generators (which run mechanisms hundreds of thousands of times to estimate their output distribution). Moreover, its sound approach also allows automatic verification of correct mechanisms. When evaluated on standard benchmarks and newer privacy mechanisms, CheckDP generates proofs (for correct mechanisms) and counterexamples (for incorrect mechanisms) within 70 seconds without any false positives or false negatives.

[1]  Ashwin Machanavajjhala,et al.  Differentially Private Algorithms for Empirical Machine Learning , 2014, ArXiv.

[2]  Úlfar Erlingsson,et al.  Prochlo: Strong Privacy for Analytics in the Crowd , 2017, SOSP.

[3]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[4]  Gilles Barthe,et al.  Probabilistic Relational Reasoning for Differential Privacy , 2012, TOPL.

[5]  Aws Albarghouthi,et al.  Synthesizing coupling proofs of differential privacy , 2017, Proc. ACM Program. Lang..

[6]  Benjamin Grégoire,et al.  Proving Differential Privacy via Probabilistic Couplings , 2016, 2016 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).

[7]  Ashwin Machanavajjhala,et al.  Privacy: Theory meets Practice on the Map , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[8]  Pedro R. D'Argenio,et al.  Secure information flow by self-composition , 2004, Proceedings. 17th IEEE Computer Security Foundations Workshop, 2004..

[9]  Joe D. Warren,et al.  The program dependence graph and its use in optimization , 1987, TOPL.

[10]  Gilles Barthe,et al.  Beyond Differential Privacy: Composition Theorems and Relational Logic for f-divergences between Probabilistic Programs , 2013, ICALP.

[11]  Úlfar Erlingsson,et al.  RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response , 2014, CCS.

[12]  Dawn Xiaodong Song,et al.  Towards Practical Differential Privacy for SQL Queries , 2017, Proc. VLDB Endow..

[13]  Janardhan Kulkarni,et al.  Collecting Telemetry Data Privately , 2017, NIPS.

[14]  Kunal Talwar,et al.  Private selection from private candidates , 2018, STOC.

[15]  Cynthia Dwork,et al.  Differential Privacy , 2006, ICALP.

[16]  Ninghui Li,et al.  Understanding the Sparse Vector Technique for Differential Privacy , 2016, Proc. VLDB Endow..

[17]  Yu Zhang,et al.  Differentially Private High-Dimensional Data Publication via Sampling-Based Inference , 2015, KDD.

[18]  Xing Xie,et al.  PrivTree: A Differentially Private Algorithm for Hierarchical Decompositions , 2016, SIGMOD Conference.

[19]  Dexter Kozen,et al.  Semantics of probabilistic programs , 1979, 20th Annual Symposium on Foundations of Computer Science (sfcs 1979).

[20]  Thomas Steinke,et al.  Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds , 2016, TCC.

[21]  Pierre-Yves Strub,et al.  Advanced Probabilistic Couplings for Differential Privacy , 2016, CCS.

[22]  Timon Gehr,et al.  PSI: Exact Symbolic Inference for Probabilistic Programs , 2016, CAV.

[23]  Ilya Mironov,et al.  Rényi Differential Privacy , 2017, 2017 IEEE 30th Computer Security Foundations Symposium (CSF).

[24]  Elaine Shi,et al.  Private and Continual Release of Statistics , 2010, TSEC.

[25]  Rohit Chadha,et al.  Deciding Differential Privacy for Programs with Finite Inputs and Outputs. , 2019 .

[26]  Danfeng Zhang,et al.  Proving differential privacy with shadow execution , 2019, PLDI.

[27]  Ashwin Machanavajjhala,et al.  On the Privacy Properties of Variants on the Sparse Vector Technique , 2015, ArXiv.

[28]  Timon Gehr,et al.  DP-Finder: Finding Differential Privacy Violations by Sampling and Optimization , 2018, CCS.

[29]  John M. Abowd,et al.  The U.S. Census Bureau Adopts Differential Privacy , 2018, KDD.

[30]  Dirk Beyer,et al.  CPAchecker: A Tool for Configurable Software Verification , 2009, CAV.

[31]  Danfeng Zhang,et al.  LightDP: towards automating differential privacy proofs , 2016, POPL.

[32]  Anna C. Gilbert,et al.  Property Testing For Differential Privacy , 2018, 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[33]  Gilles Barthe,et al.  Proving Differential Privacy in Hoare Logic , 2014, 2014 IEEE 27th Computer Security Foundations Symposium.

[34]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[35]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[36]  Chris Clifton,et al.  Top-k frequent itemsets via differentially private FP-trees , 2014, KDD.

[37]  Alexander Aiken,et al.  Secure Information Flow as a Safety Problem , 2005, SAS.

[38]  Andreas Haeberlen,et al.  Linear dependent types for differential privacy , 2013, POPL.

[39]  Danfeng Zhang,et al.  Free Gap Information from the Differentially Private Sparse Vector and Noisy Max Mechanisms , 2019, Proc. VLDB Endow..

[40]  George Danezis,et al.  Verified Computational Differential Privacy with Applications to Smart Metering , 2013, 2013 IEEE 26th Computer Security Foundations Symposium.

[41]  Moni Naor,et al.  Our Data, Ourselves: Privacy Via Distributed Noise Generation , 2006, EUROCRYPT.

[42]  Benjamin C. Pierce,et al.  Distance makes the types grow stronger: a calculus for differential privacy , 2010, ICFP '10.

[43]  Sanjit A. Seshia,et al.  Combinatorial sketching for finite programs , 2006, ASPLOS XII.

[44]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[45]  Danfeng Zhang,et al.  Detecting Violations of Differential Privacy , 2018, CCS.

[46]  Ashwin Machanavajjhala,et al.  Utility Cost of Formal Privacy for Releasing National Employer-Employee Statistics , 2017, SIGMOD Conference.

[47]  Bernard Carré,et al.  Information-flow and data-flow analysis of while-programs , 1985, TOPL.