On the Sample Complexity of Privately Learning Axis-Aligned Rectangles

We revisit the fundamental problem of learning Axis-Aligned-Rectangles over a finite grid X ⊆ R with differential privacy. Existing results show that the sample complexity of this problem is at most min { d· log |X| , d· (log∗ |X|) } . That is, existing constructions either require sample complexity that grows linearly with log |X|, or else it grows super linearly with the dimension d. We present a novel algorithm that reduces the sample complexity to only Õ ( d· (log∗ |X|) ) , attaining a dimensionality optimal dependency without requiring the sample complexity to grow with log |X|. The technique used in order to attain this improvement involves the deletion of “exposed” data-points on the go, in a fashion designed to avoid the cost of the adaptive composition theorems. The core of this technique may be of individual interest, introducing a new method for constructing statistically-efficient private algorithms.

[1]  Kobbi Nissim,et al.  Differentially Private Release and Learning of Threshold Functions , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[2]  Haim Kaplan,et al.  Private Learning of Halfspaces: Simplifying the Construction and Reducing the Sample Complexity , 2020, NeurIPS.

[3]  Guy N. Rothblum,et al.  Boosting and Differential Privacy , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[4]  Amos Beimel,et al.  Learning Privately with Labeled and Unlabeled Examples , 2014, Algorithmica.

[5]  Haim Kaplan,et al.  Differentially Private Learning of Geometric Concepts , 2019, ICML.

[6]  Kobbi Nissim,et al.  Simultaneous Private Learning of Multiple Concepts , 2015, ITCS.

[7]  Noga Alon,et al.  Private PAC learning implies finite Littlestone dimension , 2018, STOC.

[8]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[9]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[10]  Noga Alon,et al.  Closure Properties for Private Classification and Online Prediction , 2020, COLT.

[11]  Vitaly Feldman,et al.  Sample Complexity Bounds on Differentially Private Learning via Communication Complexity , 2014, SIAM J. Comput..

[12]  Roi Livni,et al.  An Equivalence Between Private Classification and Online Prediction , 2020, 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS).

[13]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[14]  Haim Kaplan,et al.  Privately Learning Thresholds: Closing the Exponential Gap , 2019, COLT.

[15]  Amos Beimel,et al.  Private Learning and Sanitization: Pure vs. Approximate Differential Privacy , 2013, APPROX-RANDOM.

[16]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[17]  Aaron Roth,et al.  Differentially private combinatorial optimization , 2009, SODA '10.

[18]  Amos Beimel,et al.  Characterizing the Sample Complexity of Pure Private Learners , 2019, J. Mach. Learn. Res..

[19]  Amos Beimel,et al.  Bounds on the sample complexity for private learning and private data release , 2010, Machine Learning.

[20]  Shay Moran,et al.  Private Center Points and Learning of Halfspaces , 2019, COLT.

[21]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[22]  Yishay Mansour,et al.  The Sparse Vector Technique, Revisited , 2020, ArXiv.

[23]  Salil P. Vadhan,et al.  The Complexity of Differential Privacy , 2017, Tutorials on the Foundations of Cryptography.

[24]  Sofya Raskhodnikova,et al.  What Can We Learn Privately? , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.