X-Secure T-Private Federated Submodel Learning

The problem of (information-theoretic) X-secure T-private federated submodel learning represents a setting where a large scale machine learning model is partitioned into K submodels and stored across N distributed servers according to an X-secure threshold secret sharing scheme. Various users wish to successively train (update) the submodel that is most relevant to their local data while keeping the identity of their relevant submodel private from any set of up to T colluding servers. Inspired by the idea of cross-subspace alignment (CSA) for X - secure T -private information retrieval, we propose a novel CSA-RW (read-write) scheme for efficiently (in communication cost) and privately reading from and writing to a distributed database. CSA-RW improves significantly upon available baselines from prior work and is shown to be asymptotically/approximately optimal in download/upload cost. It also answers an open question previously noted by Kairouz et al. by exploiting synergistic gains from the joint design of private read-write.

[1]  Shaojie Tang,et al.  Secure Federated Submodel Learning , 2019, ArXiv.

[2]  Vitaly Shmatikov,et al.  How To Backdoor Federated Learning , 2018, AISTATS.

[3]  Hua Sun,et al.  The Capacity of Private Information Retrieval , 2017, IEEE Transactions on Information Theory.

[4]  Sennur Ulukus,et al.  The Capacity of Private Information Retrieval from Decentralized Uncoded Caching Databases , 2019, Inf..

[5]  Sennur Ulukus,et al.  The Capacity of Private Information Retrieval from Byzantine and Colluding Databases , 2017, IEEE Transactions on Information Theory.

[6]  Ravi Tandon,et al.  The capacity of cache aided private information retrieval , 2017, 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[7]  Camilla Hollanti,et al.  Private information retrieval schemes for codec data with arbitrary collusion patterns , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[8]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[9]  Chao Tian,et al.  Capacity-Achieving Private Information Retrieval Codes from MDS-Coded Databases with Minimum Message Size , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[10]  Rafail Ostrovsky,et al.  Distributed Oblivious RAM for Secure Two-Party Computation , 2013, TCC.

[11]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[12]  V. Pan.  Structured Matrices and Polynomials: Unified Superfast Algorithms , 2001 .

[13]  Mahdi Jafari Siavoshani,et al.  Multi-Message Private Information Retrieval with Private Side Information , 2018, 2018 IEEE Information Theory Workshop (ITW).

[14]  Syed Ali Jafar,et al.  The Capacity of Private Information Retrieval with Private Side Information , 2017, ArXiv.

[15]  Camilla Hollanti,et al.  Private Information Retrieval from Coded Databases with Colluding Servers , 2016, SIAM J. Appl. Algebra Geom..

[16]  Wonjae Shin,et al.  Private Information Retrieval for Secure Distributed Storage Systems , 2018, IEEE Transactions on Information Forensics and Security.

[17]  Sennur Ulukus,et al.  Multi-Message Private Information Retrieval: Capacity Results and Near-Optimal Schemes , 2017, IEEE Transactions on Information Theory.

[18]  Syed Ali Jafar,et al.  On the Asymptotic Capacity of X-Secure T-Private Information Retrieval With Graph-Based Replicated Storage , 2019, IEEE Transactions on Information Theory.

[19]  Hua Sun,et al.  Private Information Retrieval from MDS Coded Data With Colluding Servers: Settling a Conjecture by Freij-Hollanti et al. , 2018, IEEE Transactions on Information Theory.

[20]  Tianjian Chen,et al.  Federated Machine Learning: Concept and Applications , 2019 .

[21]  Hua Sun,et al.  The Capacity of Private Computation , 2018, 2018 IEEE International Conference on Communications (ICC).

[22]  Syed A. Jafar,et al.  X-Secure T-Private Information Retrieval From MDS Coded Storage With Byzantine and Unresponsive Servers , 2019, IEEE Transactions on Information Theory.

[23]  Syed Ali Jafar,et al.  The Asymptotic Capacity of Private Search , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[24]  Mikael Skoglund,et al.  The Capacity of Private Information Retrieval with Eavesdroppers , 2018, 2018 52nd Asilomar Conference on Signals, Systems, and Computers.

[25]  Hua Sun,et al.  The Capacity of Symmetric Private Information Retrieval , 2016, 2016 IEEE Globecom Workshops (GC Wkshps).

[26]  Eyal Kushilevitz,et al.  Sub-logarithmic Distributed Oblivious RAM with Small Block Size , 2019, IACR Cryptol. ePrint Arch..

[27]  Sennur Ulukus,et al.  Private Set Intersection: A Multi-Message Symmetric Private Information Retrieval Perspective , 2021, IEEE Transactions on Information Theory.

[28]  Georg Heinig,et al.  An inversion formula and fast algorithms for Cauchy-Vandermonde matrices , 1993 .

[29]  M. Gasca,et al.  Computation of rational interpolants with prescribed poles , 1989 .

[30]  Hua Sun,et al.  Cross Subspace Alignment and the Asymptotic Capacity of $X$ -Secure $T$ -Private Information Retrieval , 2018, IEEE Transactions on Information Theory.

[31]  Mikael Skoglund,et al.  Secure Private Information Retrieval from Colluding Databases with Eavesdroppers , 2017, 2018 IEEE International Symposium on Information Theory (ISIT).

[32]  Mikael Skoglund,et al.  Linear symmetric private information retrieval for MDS coded distributed storage with colluding servers , 2017, 2017 IEEE Information Theory Workshop (ITW).

[33]  Sennur Ulukus,et al.  Fundamental Limits of Cache-Aided Private Information Retrieval With Unknown and Uncoded Prefetching , 2017, IEEE Transactions on Information Theory.

[34]  Deepak Kumar,et al.  The Capacity of Private Information Retrieval From Uncoded Storage Constrained Databases , 2018, IEEE Transactions on Information Theory.

[35]  Michael Gastpar,et al.  Single-server Multi-user Private Information Retrieval with Side Information , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[36]  Eyal Kushilevitz,et al.  Private information retrieval , 1998, JACM.

[37]  Hua Sun,et al.  The Capacity of Symmetric Private Information Retrieval , 2019, IEEE Transactions on Information Theory.

[38]  Hua Sun,et al.  Multiround Private Information Retrieval: Capacity and Storage Overhead , 2016, IEEE Transactions on Information Theory.

[39]  Syed A. Jafar,et al.  Double Blind T-Private Information Retrieval , 2020, IEEE Journal on Selected Areas in Information Theory.

[40]  Jungwoo Lee,et al.  Information-Theoretic Privacy in Federated Submodel learning , 2020, ICT Express.

[41]  Mikael Skoglund,et al.  The $\epsilon$-error Capacity of Symmetric PIR with Byzantine Adversaries , 2018 .

[42]  Vitaly Feldman,et al.  Privacy Amplification by Iteration , 2018, 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS).

[43]  H. Brendan McMahan,et al.  Learning Differentially Private Recurrent Language Models , 2017, ICLR.

[44]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2021, Found. Trends Mach. Learn..

[45]  Nan Liu,et al.  The Capacity of Multi-round Private Information Retrieval from Byzantine Databases , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[46]  Chao Tian,et al.  Capacity-Achieving Private Information Retrieval Codes With Optimal Message Size and Upload Cost , 2018, IEEE Transactions on Information Theory.

[47]  Camilla Hollanti,et al.  Private Information Retrieval From Coded Storage Systems With Colluding, Byzantine, and Unresponsive Servers , 2018, IEEE Transactions on Information Theory.

[48]  Hua Sun,et al.  The Capacity of Robust Private Information Retrieval With Colluding Databases , 2016, IEEE Transactions on Information Theory.

[49]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[50]  Sennur Ulukus,et al.  The Capacity of Private Information Retrieval With Partially Known Private Side Information , 2019, IEEE Transactions on Information Theory.

[51]  Mahtab Mirmohseni,et al.  Private function retrieval , 2017, 2018 Iran Workshop on Communication and Information Theory (IWCIT).

[52]  Mikael Skoglund,et al.  The ϵ-error Capacity of Symmetric PIR with Byzantine Adversaries , 2018, 2018 IEEE Information Theory Workshop (ITW).

[53]  Sennur Ulukus,et al.  The Capacity of Private Information Retrieval From Heterogeneous Uncoded Caching Databases , 2019, IEEE Transactions on Information Theory.