Federated Over-the-Air Subspace Learning and Tracking from Incomplete Data.

We consider a federated learning scenario where $K$ peer nodes communicate with a master node via a wireless channel using the newly developed ``over-the-air'' superposition and broadcast paradigm. This means that (i) data transmitted from the nodes is directly summed at the master node using the superposition property of the wireless channel; and (ii) the master broadcasts this sum, or a processed version of it, to all the nodes. The implicit assumption here is that the aggregation to be performed at the master node is an additive operation. This new transmission mode is enabled by advances in wireless technology that allow for synchronous transmission by the $K$ peer nodes. It is $K$ times time- or bandwidth- efficient compared to the traditional digital transmission mode, but the tradeoff is that channel noise corrupts each iterate of the underlying ML algorithm being implemented. Additive noise in each algorithm iterate is a completely different type of perturbation than noise or outliers in the observed data. It introduces a novel set of challenges that have not been previously explored in the literature. In this work, we develop and analyze federated over-the-air solutions to two well-studied problems in unsupervised learning: (i) subspace learning and (ii) subspace tracking from incomplete data.

[1]  Kin K. Leung,et al.  Adaptive Federated Learning in Resource Constrained Edge Computing Systems , 2018, IEEE Journal on Selected Areas in Communications.

[2]  W. Kahan,et al.  The Rotation of Eigenvectors by a Perturbation. III , 1970 .

[3]  M. Rudelson,et al.  The smallest singular value of a random rectangular matrix , 2008, 0802.3956.

[4]  Namrata Vaswani,et al.  Provable Subspace Tracking From Missing Data and Matrix Completion , 2018, IEEE Transactions on Signal Processing.

[5]  Jon Crowcroft,et al.  Federated Principal Component Analysis , 2019, NeurIPS.

[6]  Anna Scaglione,et al.  A Review of Distributed Algorithms for Principal Component Analysis , 2018, Proceedings of the IEEE.

[7]  Namrata Vaswani,et al.  NEARLY OPTIMAL ROBUST SUBSPACE TRACKING: A UNIFIED APPROACH , 2017, 2018 IEEE Data Science Workshop (DSW).

[8]  Qiang Yang,et al.  Federated Machine Learning , 2019, ACM Trans. Intell. Syst. Technol..

[9]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..

[10]  Peter Richtárik,et al.  Federated Optimization: Distributed Machine Learning for On-Device Intelligence , 2016, ArXiv.

[11]  Namrata Vaswani,et al.  PCA in Sparse Data-Dependent Noise , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[12]  Deniz Gündüz,et al.  Federated Learning Over Wireless Fading Channels , 2019, IEEE Transactions on Wireless Communications.

[13]  Prateek Jain,et al.  Low-rank matrix completion using alternating minimization , 2012, STOC '13.

[14]  Dejiao Zhang,et al.  Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation , 2015, AISTATS.

[15]  A. Robert Calderbank,et al.  PETRELS: Parallel Subspace Estimation and Tracking by Recursive Least Squares From Partial Observations , 2012, IEEE Transactions on Signal Processing.

[16]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[17]  Mihaela van der Schaar,et al.  Machine Learning in the Air , 2019, IEEE Journal on Selected Areas in Communications.

[18]  Dan Alistarh,et al.  Byzantine Stochastic Gradient Descent , 2018, NeurIPS.

[19]  Benjamin Recht,et al.  A Simpler Approach to Matrix Completion , 2009, J. Mach. Learn. Res..

[20]  Qing Ling,et al.  Byzantine-Robust Stochastic Gradient Descent for Distributed Low-Rank Matrix Completion , 2019, 2019 IEEE Data Science Workshop (DSW).

[21]  Yonina C. Eldar,et al.  Subspace Learning with Partial Information , 2014, J. Mach. Learn. Res..

[22]  Indranil Gupta,et al.  Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation , 2019, UAI.

[23]  Moritz Hardt,et al.  The Noisy Power Method: A Meta Algorithm with Applications , 2013, NIPS.

[24]  Zhi Ding,et al.  Federated Learning via Over-the-Air Computation , 2018, IEEE Transactions on Wireless Communications.