Towards Sharper Utility Bounds for Differentially Private Pairwise Learning

Pairwise learning focuses on learning tasks with pairwise loss functions, depends on pairs of training instances, and naturally fits for modeling relationships between pairs of samples. In this paper, we focus on the privacy of pairwise learning and propose a new differential privacy paradigm for pairwise learning, based on gradient perturbation. Except for the privacy guarantees, we also analyze the excess population risk and give corresponding bounds under both expectation and high probability conditions. We use the on-average stability and the pairwise locally elastic stability theories to analyze the expectation bound and the high probability bound, respectively. Moreover, our analyzed utility bounds do not require convex pairwise loss functions, which means that our method is general to both convex and non-convex conditions. Under these circumstances, the utility bounds are similar to (or better than) previous bounds under convexity or strongly convexity assumption, which are attractive results.

[1]  Di Wang,et al.  Differentially Private Empirical Risk Minimization with Non-convex Loss Functions , 2019, ICML.

[2]  Kamalika Chaudhuri,et al.  Privacy-preserving logistic regression , 2008, NIPS.

[3]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[4]  Zhe-Ming Lu,et al.  Face recognition based on local binary pattern and improved Pairwise-constrained Multiple Metric Learning , 2019, Multimedia Tools and Applications.

[5]  S. Linnainmaa.  Taylor expansion of the accumulated rounding error , 1976 .

[6]  Percy Liang,et al.  Understanding Black-box Predictions via Influence Functions , 2017, ICML.

[7]  Di Wang,et al.  Differentially Private Empirical Risk Minimization Revisited: Faster and More General , 2018, NIPS.

[8]  Matus Telgarsky,et al.  Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis , 2017, COLT.

[9]  Mário A. T. Figueiredo,et al.  A Classification-Based Approach to Semi-Supervised Clustering with Pairwise Constraints , 2020, Neural Networks.

[10]  G. Lugosi,et al.  Ranking and empirical minimization of U-statistics , 2006, math/0603123.

[11]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[12]  Mark W. Schmidt,et al.  Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition , 2016, ECML/PKDD.

[13]  Dacheng Tao,et al.  Algorithmic Stability and Hypothesis Complexity , 2017, ICML.

[14]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[15]  Ke Tang,et al.  Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Siwei Lyu,et al.  Stability and Differential Privacy of Stochastic Gradient Descent for Pairwise Learning with Non-Smooth Loss , 2021, AISTATS.

[17]  M. Tahar Kechadi,et al.  Combining Real-Time Segmentation and Classification of Rehabilitation Exercises with LSTM Networks and Pointwise Boosting , 2020, AAAI.

[18]  Hangfeng He,et al.  Toward Better Generalization Bounds with Locally Elastic Stability , 2020, ICML.

[19]  Stefan Kramer,et al.  Fair pairwise learning to rank , 2020, 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA).

[20]  Qiong Cao,et al.  Generalization bounds for metric and similarity learning , 2012, Machine Learning.

[21]  Raef Bassily,et al.  Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds , 2014, 1405.7085.

[22]  Somesh Jha,et al.  Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing , 2014, USENIX Security Symposium.

[23]  Marc Sebban,et al.  A Survey on Metric Learning for Feature Vectors and Structured Data , 2013, ArXiv.

[24]  Daniel Kifer,et al.  Private Convex Empirical Risk Minimization and High-dimensional Regression , 2012, COLT 2012.

[25]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[26]  Liwei Wang,et al.  Efficient Private ERM for Smooth Objectives , 2017, IJCAI.

[27]  Anand D. Sarwate,et al.  Differentially Private Empirical Risk Minimization , 2009, J. Mach. Learn. Res..

[28]  Chenglin Miao,et al.  Pairwise Learning with Differential Privacy Guarantees , 2020, AAAI.

[29]  Marius Kloft,et al.  Sharper Generalization Bounds for Pairwise Learning , 2020, NeurIPS.

[30]  Shivani Agarwal,et al.  Generalization Bounds for Ranking Algorithms via Algorithmic Stability , 2009, J. Mach. Learn. Res..