Private Learning of Halfspaces: Simplifying the Construction and Reducing the Sample Complexity

We present a differentially private learner for halfspaces over a finite grid $G$ in $\mathbb{R}^d$ with sample complexity $\approx d^{2.5}\cdot 2^{\log^*|G|}$, which improves the state-of-the-art result of [Beimel et al., COLT 2019] by a $d^2$ factor. The building block for our learner is a new differentially private algorithm for approximately solving the linear feasibility problem: Given a feasible collection of $m$ linear constraints of the form $Ax\geq b$, the task is to privately identify a solution $x$ that satisfies most of the constraints. Our algorithm is iterative, where each iteration determines the next coordinate of the constructed solution $x$.

[1]  Tim Roughgarden,et al.  Privately Solving Linear Programs , 2014, ICALP.

[2]  Shay Moran,et al.  Private Center Points and Learning of Halfspaces , 2019, COLT.

[3]  Thomas Steinke,et al.  Composable and versatile privacy via truncated CDP , 2018, STOC.

[4]  Noga Alon,et al.  Private PAC learning implies finite Littlestone dimension , 2018, STOC.

[5]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[6]  Guy N. Rothblum,et al.  Boosting and Differential Privacy , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[7]  Santosh S. Vempala,et al.  A simple polynomial-time rescaling algorithm for solving linear programs , 2004, STOC '04.

[8]  Kobbi Nissim,et al.  Differentially Private Release and Learning of Threshold Functions , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[9]  Huy L. Nguyen,et al.  Efficient Private Algorithms for Learning Halfspaces , 2019, ArXiv.

[10]  Vitaly Feldman,et al.  Sample Complexity Bounds on Differentially Private Learning via Communication Complexity , 2014, SIAM J. Comput..

[11]  Michael Kearns,et al.  Efficient noise-tolerant learning from statistical queries , 1993, STOC.

[12]  Haim Kaplan,et al.  Privately Learning Thresholds: Closing the Exponential Gap , 2019, COLT.

[13]  Amos Beimel,et al.  Private Learning and Sanitization: Pure vs. Approximate Differential Privacy , 2013, APPROX-RANDOM.

[14]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[15]  Moni Naor,et al.  Our Data, Ourselves: Privacy Via Distributed Noise Generation , 2006, EUROCRYPT.

[16]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[17]  Cynthia Dwork,et al.  Differential privacy and robust statistics , 2009, STOC '09.

[18]  Kunal Talwar,et al.  Mechanism Design via Differential Privacy , 2007, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[19]  Cynthia Dwork,et al.  Practical privacy: the SuLQ framework , 2005, PODS.

[20]  Sofya Raskhodnikova,et al.  What Can We Learn Privately? , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[21]  Adam D. Smith,et al.  Differentially Private Feature Selection via Stability Arguments, and the Robustness of the Lasso , 2013, COLT.

[22]  Amos Beimel,et al.  Characterizing the sample complexity of private learners , 2013, ITCS '13.

[23]  Roi Livni,et al.  An Equivalence Between Private Classification and Online Prediction , 2020, 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS).

[24]  Anand D. Sarwate,et al.  Differentially Private Empirical Risk Minimization , 2009, J. Mach. Learn. Res..

[25]  Raef Bassily,et al.  Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds , 2014, 1405.7085.