Protea: client profiling within federated systems using flower

Federated Learning (FL) has emerged as a prospective solution that facilitates the training of a high-performing centralised model without compromising the privacy of users. While successful, FL research is currently limited by the difficulties of establishing a realistic large-scale FL system at the early stages of experimentation. Simulation can help accelerate this process. To facilitate efficient scalable FL simulation of heterogeneous clients, we design and implement Protea, a flexible and lightweight client profiling component within federated systems using the FL framework Flower. It allows automatically collecting system-level statistics and estimating the resources needed for each client, thus running the simulation in a resource-aware fashion. The results show that our design successfully increases parallelism for 1.66 X faster wall-clock time and 2.6X better GPU utilisation, which enables large-scale experiments on heterogeneous clients.

[1]  Titouan Parcollet,et al.  ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity , 2022, ICLR.

[2]  Yasar Abbas Ur Rehman,et al.  Federated Self-supervised Learning for Video Understanding , 2022, ECCV.

[3]  Andre Manoel,et al.  FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations , 2022, ArXiv.

[4]  Yonggang Wen,et al.  EasyFL: A Low-Code Federated Learning Platform for Dummies , 2021, IEEE Internet of Things Journal.

[5]  Daniel J. Beutel,et al.  End-to-End Speech Recognition from Federated Acoustic Models , 2021, ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Daniel J. Beutel,et al.  Secure aggregation for federated learning in flower , 2021, DistributedML@CoNEXT.

[7]  Ananda Theertha Suresh,et al.  FedJAX: Federated learning simulation with JAX , 2021, ArXiv.

[8]  Sanjay Sri Vallabh Singapuram,et al.  FedScale: Benchmarking Model and System Performance of Federated Learning at Scale , 2021, ICML.

[9]  Micah J. Sheller,et al.  OpenFL: the open federated learning library , 2021, Physics in medicine and biology.

[10]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[11]  Jonathan Passerat-Palmbach,et al.  PySyft: A Library for Easy Federated Learning , 2021 .

[12]  Daniel J. Beutel,et al.  Flower: A Friendly Federated Learning Research Framework , 2020, 2007.14390.

[13]  Ramesh Raskar,et al.  FedML: A Research Library and Benchmark for Federated Machine Learning , 2020, ArXiv.

[14]  Lalana Kagal,et al.  PrivacyFL: A Simulator for Privacy-Preserving and Secure Federated Learning , 2020, CIKM.

[15]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[16]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[17]  Sebastian Caldas,et al.  LEAF: A Benchmark for Federated Settings , 2018, ArXiv.

[18]  Michael I. Jordan,et al.  Ray: A Distributed Framework for Emerging AI Applications , 2017, OSDI.

[19]  Gregory Cohen,et al.  EMNIST: Extending MNIST to handwritten letters , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[20]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[21]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Zheng Zhang,et al.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems , 2015, ArXiv.

[24]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .