FedEval: A Holistic Evaluation Framework for Federated Learning

—Federated Learning (FL) has been widely accepted as the solution for privacy-preserving machine learning without collecting raw data. While new technologies proposed in the past few years do evolve the FL area, unfortunately, the evaluation results presented in these works fall short in integrity and are hardly comparable because of the inconsistent evaluation metrics and experimental settings. In this paper, we propose a holistic evaluation framework for FL called FedEval, and present a benchmarking study on seven state-of-the-art FL algorithms. Specifically, we first introduce the core evaluation taxonomy model, called FedEval-Core, which covers four essential evaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency, with various well-defined metrics and experimental settings. Based on the FedEval-Core, we further develop an FL evaluation platform with standardized evaluation settings and easy-to-use interfaces. We then provide an in-depth benchmarking study between the seven well-known FL algorithms, including FedSGD [1], FedAvg [1], FedProx [2], FedOpt [3], FedSTC [4], SecAgg [5], and HEAgg [6, 7]. We comprehensively analyze the advantages and disadvantages of these algorithms and further identify the suitable practical scenarios for different algorithms, which is rarely done by prior work. Lastly, we excavate a set of take-away insights and future research directions, which are very helpful for researchers in the FL area.

[1]  Yang Qin,et al.  A state-of-the-art survey on solving non-IID data in Federated Learning , 2022, Future Gener. Comput. Syst..

[2]  Qiang Yang,et al.  No Free Lunch Theorem for Security and Utility in Federated Learning , 2022, ACM Trans. Intell. Syst. Technol..

[3]  Venkatesh Saligrama,et al.  Federated Learning Based on Dynamic Regularization , 2021, ICLR.

[4]  Eunho Yang,et al.  FedMix: Approximation of Mixup under Mean Augmented Federated Learning , 2021, ICLR.

[5]  Sanjay Sri Vallabh Singapuram,et al.  FedScale: Benchmarking Model and System Performance of Federated Learning at Scale , 2021, ICML.

[6]  Junxue Zhang,et al.  Practical Lossless Federated Singular Vector Decomposition over Billion-Scale Data , 2021, KDD.

[7]  Yann Fraboni,et al.  Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning , 2021, ICML.

[8]  Ethan Fetaya,et al.  Personalized Federated Learning using Hypernetworks , 2021, ICML.

[9]  Virginia Smith,et al.  Heterogeneity for the Win: One-Shot Federated Clustering , 2021, ICML.

[10]  Mehrdad Mahdavi,et al.  Distributionally Robust Federated Averaging , 2021, NeurIPS.

[11]  Q. Dou,et al.  FedBN: Federated Learning on Non-IID Features via Local Batch Normalization , 2021, ICLR.

[12]  S. Shakkottai,et al.  Exploiting Shared Representations for Personalized Federated Learning , 2021, ICML.

[13]  P. Kairouz,et al.  The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation , 2021, ICML.

[14]  Tianbao Yang,et al.  Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity , 2021, ICML.

[15]  Taiji Suzuki,et al.  Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning , 2021, ICML.

[16]  Haibo Yang,et al.  Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning , 2021, ICLR.

[17]  Sanja Fidler,et al.  Personalized Federated Learning with First Order Model Optimization , 2020, ICLR.

[18]  G. Neglia,et al.  Throughput-Optimal Topology Design for Cross-Silo Federated Learning , 2020, NeurIPS.

[19]  Abhimanyu Dubey,et al.  Differentially-Private Federated Linear Bandits , 2020, NeurIPS.

[20]  Kian Hsiang Low,et al.  Federated Bayesian Optimization via Thompson Sampling , 2020, NeurIPS.

[21]  M. Chowdhury,et al.  Oort: Efficient Federated Learning via Guided Participant Selection , 2020, OSDI.

[22]  E. Xing,et al.  Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms , 2020, ICLR.

[23]  Filip Hanzely,et al.  Lower Bounds and Optimal Algorithms for Personalized Federated Learning , 2020, NeurIPS.

[24]  Jie Ding,et al.  HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients , 2020, ICLR.

[25]  Shaojie Tang,et al.  Billion-scale federated learning on mobile clients: a submodel design with tunable privacy , 2020, MobiCom.

[26]  Hong-You Chen,et al.  FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning , 2020, ICLR.

[27]  Jean-Pierre Hubaux,et al.  POSEIDON: Privacy-Preserving Federated Neural Network Learning , 2020, NDSS.

[28]  Murali Annavaram,et al.  Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge , 2020, NeurIPS.

[29]  Daniel J. Beutel,et al.  Flower: A Friendly Federated Learning Research Framework , 2020, 2007.14390.

[30]  Ramesh Raskar,et al.  FedML: A Research Library and Benchmark for Federated Machine Learning , 2020, ArXiv.

[31]  Vladimir Braverman,et al.  FetchSGD: Communication-Efficient Federated Learning with Sketching , 2022 .

[32]  Qinghua Liu,et al.  Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization , 2020, NeurIPS.

[33]  Ananda Theertha Suresh,et al.  FedBoost: A Communication-Efficient Algorithm for Federated Learning , 2020, ICML.

[34]  Xiang Li,et al.  Federated Doubly Stochastic Kernel Learning for Vertically Partitioned Data , 2020, KDD.

[35]  Eunho Yang,et al.  Federated Semi-Supervised Learning with Inter-Client Consistency , 2020, ArXiv.

[36]  Ali Jadbabaie,et al.  Robust Federated Learning: The Case of Affine Distribution Shifts , 2020, NeurIPS.

[37]  Tengyu Ma,et al.  Federated Accelerated Stochastic Gradient Descent , 2020, NeurIPS.

[38]  Bingsheng He,et al.  The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems , 2020, ACM Trans. Intell. Syst. Technol..

[39]  Sebastian U. Stich,et al.  Ensemble Distillation for Robust Model Fusion in Federated Learning , 2020, NeurIPS.

[40]  K. Ramchandran,et al.  An Efficient Framework for Clustered Federated Learning , 2020, IEEE Transactions on Information Theory.

[41]  Nguyen H. Tran,et al.  Personalized Federated Learning with Moreau Envelopes , 2020, NeurIPS.

[42]  Shui Yu,et al.  GAN Enhanced Membership Inference: A Passive Local Attack in Federated Learning , 2020, ICC 2020 - 2020 IEEE International Conference on Communications (ICC).

[43]  Martin J. Wainwright,et al.  FedSplit: An algorithmic framework for fast federated optimization , 2020, NeurIPS.

[44]  Sanjiv Kumar,et al.  Federated Learning with Only Positive Labels , 2020, ICML.

[45]  Peter Richtárik,et al.  From Local SGD to Local Fixed Point Methods for Federated Learning , 2020, ICML.

[46]  Jun Xiao,et al.  Evaluation Framework For Large-scale Federated Learning , 2020, ArXiv.

[47]  Manzil Zaheer,et al.  Adaptive Federated Optimization , 2020, ICLR.

[48]  Zhize Li,et al.  Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization , 2020, ICML.

[49]  Yasaman Khazaeni,et al.  Federated Learning with Matched Averaging , 2020, ICLR.

[50]  Kate Saenko,et al.  Federated Adversarial Domain Adaptation , 2019, ICLR.

[51]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning , 2019, ICML.

[52]  Jon Crowcroft,et al.  Federated Principal Component Analysis , 2019, NeurIPS.

[53]  Kai Chen,et al.  Secure Federated Matrix Factorization , 2019, IEEE Intelligent Systems.

[54]  Song Han,et al.  Deep Leakage from Gradients , 2019, NeurIPS.

[55]  Shiho Moriai,et al.  Privacy-Preserving Deep Learning via Additively Homomorphic Encryption , 2019, 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH).

[56]  Tian Li,et al.  Fair Resource Allocation in Federated Learning , 2019, ICLR.

[57]  Yasaman Khazaeni,et al.  Bayesian Nonparametric Federated Learning of Neural Networks , 2019, ICML.

[58]  Klaus-Robert Müller,et al.  Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[59]  Tianjian Chen,et al.  Federated Machine Learning: Concept and Applications , 2019 .

[60]  Mehryar Mohri,et al.  Agnostic Federated Learning , 2019, ICML.

[61]  Anit Kumar Sahu,et al.  Federated Optimization in Heterogeneous Networks , 2018, MLSys.

[62]  Mats Jirstrand,et al.  A Performance Evaluation of Federated Learning Algorithms , 2018, DIDL@Middleware.

[63]  Sebastian Caldas,et al.  LEAF: A Benchmark for Federated Settings , 2018, ArXiv.

[64]  Goichiro Hanaoka,et al.  Model Inversion Attacks for Online Prediction Systems: Without Knowledge of Non-Sensitive Attributes , 2018, IEICE Trans. Inf. Syst..

[65]  Vitaly Shmatikov,et al.  How To Backdoor Federated Learning , 2018, AISTATS.

[66]  Yue Zhao,et al.  Federated Learning with Non-IID Data , 2018, ArXiv.

[67]  Sarvar Patel,et al.  Practical Secure Aggregation for Privacy-Preserving Machine Learning , 2017, IACR Cryptol. ePrint Arch..

[68]  Ameet Talwalkar,et al.  Federated Multi-Task Learning , 2017, NIPS.

[69]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[70]  Somesh Jha,et al.  Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures , 2015, CCS.

[71]  Qiang Yang,et al.  FATE: An Industrial Grade Platform for Collaborative Learning With Data Protection , 2021, J. Mach. Learn. Res..

[72]  Baihe Huang,et al.  FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Analysis , 2021, ICML.

[73]  Venkatesh Saligrama,et al.  Debiasing Model Updates for Improving Personalized Federated Training , 2021, ICML.

[74]  Aryan Mokhtari,et al.  Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach , 2020, NeurIPS.

[75]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.