FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks

Federated Learning (FL) is a distributed learning paradigm that can learn a global or personalized model from decentralized datasets on edge devices. However, in the computer vision domain, model performance in FL is far behind centralized training due to the lack of exploration in diverse tasks with a unified FL framework. FL has rarely been demonstrated effectively in advanced computer vision tasks such as object detection and image segmentation. To bridge the gap and facilitate the development of FL for computer vision tasks, in this work, we propose a federated learning library and benchmarking framework, named FedCV, to evaluate FL on the three most representative computer vision tasks: image classification, image segmentation, and object detection. We provide non-I.I.D. benchmarking datasets, models, and various reference FL algorithms. Our benchmark study suggests that there are multiple challenges that deserve future exploration: centralized training tricks may not be directly applied to FL; the non-I.I.D. dataset actually downgrades the model accuracy to some degree in different tasks; improving the system efficiency of federated training is challenging given the huge number of parameters and the per-client memory cost. We believe that such a library and benchmark, along with comparable evaluation settings, is necessary to make meaningful progress in FL on computer vision tasks. FedCV is publicly available: https://github.com/FedML-AI/FedCV .

[1]  Jian Pei,et al.  Personalized Cross-Silo Federated Learning on Non-IID Data , 2020, AAAI.

[2]  Boi Faltings,et al.  Federated Learning with Bayesian Differential Privacy , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[3]  Xinlei Chen,et al.  Exploring Simple Siamese Representation Learning , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[5]  Amir Salman Avestimehr,et al.  Mitigating Byzantine Attacks in Federated Learning , 2020, ArXiv.

[6]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[7]  Amir Salman Avestimehr,et al.  PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models , 2021, ICML.

[8]  A. Salman Avestimehr,et al.  Byzantine-Resilient Secure Federated Learning , 2020, IEEE Journal on Selected Areas in Communications.

[9]  Yunfeng Liu,et al.  Federated Object Detection: Optimizing Object Detection Model with Federated Learning , 2019, ICVISP.

[10]  Tzu-Ming Harry Hsu,et al.  Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification , 2019, ArXiv.

[11]  Mladen Kolar,et al.  Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques , 2021, ArXiv.

[12]  Daguang Xu,et al.  Privacy-preserving Federated Brain Tumour Segmentation , 2019, MLMI@MICCAI.

[13]  Yasaman Khazaeni,et al.  Bayesian Nonparametric Federated Learning of Neural Networks , 2019, ICML.

[14]  Daniel Rueckert,et al.  A generic framework for privacy preserving deep learning , 2018, ArXiv.

[15]  Y. Mansour,et al.  Three Approaches for Personalization with Applications to Federated Learning , 2020, ArXiv.

[16]  Yasaman Khazaeni,et al.  Federated Learning with Matched Averaging , 2020, ICLR.

[17]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[18]  Tassilo Klein,et al.  Differentially Private Federated Learning: A Client Level Perspective , 2017, ArXiv.

[19]  Amir Salman Avestimehr,et al.  FairFed: Enabling Group Fairness in Federated Learning , 2021, ArXiv.

[20]  Ananda Theertha Suresh,et al.  Can You Really Backdoor Federated Learning? , 2019, ArXiv.

[21]  Songze Li,et al.  OmniLytics: A Blockchain-based Secure Data Market for Decentralized Machine Learning , 2021, IACR Cryptol. ePrint Arch..

[22]  Nguyen H. Tran,et al.  Personalized Federated Learning with Moreau Envelopes , 2020, NeurIPS.

[23]  Tzu-Ming Harry Hsu,et al.  Federated Visual Classification with Real-World Data Distribution , 2020, ECCV.

[24]  Leana Golubchik,et al.  Backdoor Attacks on Federated Meta-Learning , 2020, ArXiv.

[25]  Tianjian Chen,et al.  FedVision: An Online Visual Object Detection Platform Powered by Federated Learning , 2020, AAAI.

[26]  Suhas Diggavi,et al.  A Field Guide to Federated Optimization , 2021, ArXiv.

[27]  Georg Heigold,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2021, ICLR.

[28]  Amir Salman Avestimehr,et al.  SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision , 2021, ArXiv.

[29]  Vladimir Braverman,et al.  FetchSGD: Communication-Efficient Federated Learning with Sketching , 2020, ICML.

[30]  Xiang Ren,et al.  FedNLP: A Research Platform for Federated Learning in Natural Language Processing , 2021, ArXiv.

[31]  Quoc V. Le,et al.  AutoAugment: Learning Augmentation Policies from Data , 2018, ArXiv.

[32]  Ramesh Raskar,et al.  FedML: A Research Library and Benchmark for Federated Machine Learning , 2020, ArXiv.

[33]  Sanjiv Kumar,et al.  Federated Learning with Only Positive Labels , 2020, ICML.

[34]  Dimitris N. Metaxas,et al.  Synthetic Learning: Learn From Distributed Asynchronized Discriminator GAN Without Sharing Medical Image Data , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Chaoyang He,et al.  Layer-wise Adaptive Model Aggregation for Scalable Federated Learning , 2021, ArXiv.

[36]  Xun Qian,et al.  Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization , 2020, ICML.

[37]  Alejandro F. Frangi,et al.  Fed-Sim: Federated Simulation for Medical Imaging , 2020, MICCAI.

[38]  Chaoyang He,et al.  SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks , 2021, ArXiv.

[39]  Amir Salman Avestimehr,et al.  LightSecAgg: Rethinking Secure Aggregation in Federated Learning , 2021, ArXiv.

[40]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Ramesh Raskar,et al.  Split learning for health: Distributed deep learning without sharing raw patient data , 2018, ArXiv.

[42]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[43]  Bo Li,et al.  DBA: Distributed Backdoor Attacks against Federated Learning , 2020, ICLR.

[44]  Xiang Li,et al.  On the Convergence of FedAvg on Non-IID Data , 2019, ICLR.

[45]  Laurent Condat,et al.  From Local SGD to Local Fixed Point Methods for Federated Learning , 2020, ICML.

[46]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning , 2019, ICML.

[47]  A. Salman Avestimehr,et al.  Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning , 2020, IEEE Journal on Selected Areas in Information Theory.

[48]  Yanzhao Wu,et al.  A Framework for Evaluating Gradient Leakage Attacks in Federated Learning , 2020, ArXiv.

[49]  Tobias Weyand,et al.  Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Murali Annavaram,et al.  Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge , 2020, NeurIPS.

[51]  Mehryar Mohri,et al.  Agnostic Federated Learning , 2019, ICML.

[52]  Runhua Xu,et al.  HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning , 2019, AISec@CCS.

[53]  Ivan Beschastnikh,et al.  Mitigating Sybils in Federated Learning Poisoning , 2018, ArXiv.

[54]  Seong Joon Oh,et al.  Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning , 2018, 1805.05838.

[55]  Amir Salman Avestimehr,et al.  FedNAS: Federated Deep Learning via Neural Architecture Search , 2020, ArXiv.

[56]  Qinghua Liu,et al.  Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization , 2020, NeurIPS.

[57]  Vitaly Shmatikov,et al.  How To Backdoor Federated Learning , 2018, AISTATS.

[58]  Xiaolong Ma,et al.  MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge , 2021, NeurIPS.

[59]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[60]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2021, Found. Trends Mach. Learn..

[61]  Karan Singhal,et al.  Federated Reconstruction: Partially Local Federated Learning , 2021, ArXiv.

[62]  Amir Salman Avestimehr,et al.  Federated Learning for Internet of Things , 2021, SenSys.

[63]  Boi Faltings,et al.  Federated Generative Privacy , 2019, IEEE Intelligent Systems.

[64]  Ramesh Raskar,et al.  Distributed learning of deep neural network over multiple agents , 2018, J. Netw. Comput. Appl..

[65]  Vitaly Shmatikov,et al.  Salvaging Federated Learning by Local Adaptation , 2020, ArXiv.

[66]  Rui Zhang,et al.  A Hybrid Approach to Privacy-Preserving Federated Learning , 2018, Informatik Spektrum.

[67]  Yang Song,et al.  Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning , 2018, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[68]  Anit Kumar Sahu,et al.  On the Convergence of Federated Optimization in Heterogeneous Networks , 2018, ArXiv.

[69]  Vitaly Shmatikov,et al.  Exploiting Unintended Feature Leakage in Collaborative Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[70]  Quoc V. Le,et al.  Searching for MobileNetV3 , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[71]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[72]  Sarvar Patel,et al.  Practical Secure Aggregation for Federated Learning on User-Held Data , 2016, ArXiv.

[73]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[74]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[75]  Haishan Ye,et al.  MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Tianjian Chen,et al.  Federated Machine Learning: Concept and Applications , 2019 .

[77]  Zaid Al-Ars,et al.  An Overview of Federated Deep Learning Privacy Attacks and Defensive Strategies , 2020, Federated Learning Systems.

[78]  Prateek Mittal,et al.  Analyzing Federated Learning through an Adversarial Lens , 2018, ICML.

[79]  Hanlin Tang,et al.  Central Server Free Federated Learning over Single-sided Trust Social Networks , 2019, ArXiv.

[80]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[81]  Manzil Zaheer,et al.  Adaptive Federated Optimization , 2020, ICLR.

[82]  Aryan Mokhtari,et al.  Personalized Federated Learning: A Meta-Learning Approach , 2020, ArXiv.

[83]  Bhaskar Krishnamachari,et al.  Federated Learning for Internet of Things: Applications, Challenges, and Opportunities , 2021, ArXiv.

[84]  Junzhou Huang,et al.  FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks , 2021, 2104.07145.

[85]  Tian Li,et al.  Fair Resource Allocation in Federated Learning , 2019, ICLR.

[86]  Subhransu Maji,et al.  Semantic contours from inverse detectors , 2011, 2011 International Conference on Computer Vision.