A Systematic Literature Review on Federated Learning: From A Model Quality Perspective

As an emerging technique, Federated Learning (FL) can jointly train a global model with the data remaining locally, which effectively solves the problem of data privacy protection through the encryption mechanism. The clients train their local model, and the server aggregates models until convergence. In this process, the server uses an incentive mechanism to encourage clients to contribute high-quality and large-volume data to improve the global model. Although some works have applied FL to the Internet of Things (IoT), medicine, manufacturing, etc., the application of FL is still in its infancy, and many related issues need to be solved. Improving the quality of FL models is one of the current research hotspots and challenging tasks. This paper systematically reviews and objectively analyzes the approaches to improving the quality of FL models. We are also interested in the research and application trends of FL and the effect comparison between FL and non-FL because the practitioners usually worry that achieving privacy protection needs compromising learning quality. We use a systematic review method to analyze 147 latest articles related to FL. This review provides useful information and insights to both academia and practitioners from the industry. We investigate research questions about academic research and industrial application trends of FL, essential factors affecting the quality of FL models, and compare FL and non-FL algorithms in terms of learning quality. Based on our review's conclusion, we give some suggestions for improving the FL model quality. Finally, we propose an FL application framework for practitioners.

[1]  Shengli Xie,et al.  Incentive Mechanism for Reliable Federated Learning: A Joint Optimization Approach to Combining Reputation and Contract Theory , 2019, IEEE Internet of Things Journal.

[2]  Diansheng Guo,et al.  PMF: A Privacy-preserving Human Mobility Prediction Framework via Federated Learning , 2020, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[3]  Solmaz Niknam,et al.  Federated Learning for Wireless Communications: Motivation, Opportunities, and Challenges , 2019, IEEE Communications Magazine.

[4]  Jing Xiao,et al.  Federated Learning of Unsegmented Chinese Text Recognition Model , 2019, 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI).

[5]  H. Vincent Poor,et al.  On Safeguarding Privacy and Security in the Framework of Federated Learning , 2020, IEEE Network.

[6]  Qiang Yang,et al.  Federated Reinforcement Learning , 2019, ArXiv.

[7]  Qiang Yang,et al.  SecureBoost: A Lossless Federated Learning Framework , 2019, IEEE Intelligent Systems.

[8]  Xiaoyan Sun,et al.  Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Tom Ouyang,et al.  Federated Learning Of Out-Of-Vocabulary Words , 2019, ArXiv.

[10]  Mehryar Mohri,et al.  SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning , 2019, ArXiv.

[11]  Kang G. Shin,et al.  Federated User Representation Learning , 2019, ArXiv.

[12]  Nassir Navab,et al.  BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning , 2019, ArXiv.

[13]  Yi Zhou,et al.  Towards Federated Graph Learning for Collaborative Financial Crimes Detection , 2019, ArXiv.

[14]  Ming Liu,et al.  Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data , 2019, ArXiv.

[15]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[16]  Huadong Ma,et al.  Federated Region-Learning: An Edge Computing Based Framework for Urban Environment Sensing , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[17]  Zhu Han,et al.  Incentivize to Build: A Crowdsourcing Framework for Federated Learning , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[18]  Bruno Sericola,et al.  MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets , 2018, 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[19]  Zhicong Huang,et al.  Quantification of the Leakage in Federated Learning , 2019, ArXiv.

[20]  Lifeng Sun,et al.  Towards Faster and Better Federated Learning: A Feature Fusion Approach , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[21]  Huafei Zhu,et al.  Privacy-preserving Weighted Federated Learning within Oracle-Aided MPC Framework , 2020, ArXiv.

[22]  Wei Shi,et al.  Federated learning of predictive models from federated Electronic Health Records , 2018, Int. J. Medical Informatics.

[23]  Yuntao Li,et al.  Federated Learning for Time Series Forecasting Using Hybrid Model , 2019 .

[24]  Joseph Dureau,et al.  Federated Learning for Keyword Spotting , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[25]  Yang Liu,et al.  Real-World Image Datasets for Federated Learning , 2019, ArXiv.

[26]  Chunyan Miao,et al.  Hierarchical Incentive Mechanism Design for Federated Machine Learning in Mobile Networks , 2020, IEEE Internet of Things Journal.

[27]  Han Yu,et al.  Threats to Federated Learning: A Survey , 2020, ArXiv.

[28]  Haomiao Yang,et al.  Efficient and Privacy-Enhanced Federated Learning for Industrial Artificial Intelligence , 2020, IEEE Transactions on Industrial Informatics.

[29]  Nguyen H. Tran,et al.  Federated Learning-Based Cognitive Detection of Jamming Attack in Flying Ad-Hoc Network , 2020, IEEE Access.

[30]  Dianbo Liu,et al.  Federated machine learning with Anonymous Random Hybridization (FeARH) on medical records , 2020, ArXiv.

[31]  Joachim M. Buhmann,et al.  Variational Federated Multi-Task Learning , 2019, ArXiv.

[32]  Ying-Chang Liang,et al.  Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach , 2019, 2019 IEEE VTS Asia Pacific Wireless Communications Symposium (APWCS).

[33]  Hubert Eichner,et al.  APPLIED FEDERATED LEARNING: IMPROVING GOOGLE KEYBOARD QUERY SUGGESTIONS , 2018, ArXiv.

[34]  Saraju P. Mohanty,et al.  Preserving Data Privacy via Federated Learning: Challenges and Solutions , 2020, IEEE Consumer Electronics Magazine.

[35]  Di Wu,et al.  Multi-Task Network Anomaly Detection using Federated Learning , 2019, SoICT.

[36]  Wendong Wang,et al.  Environmental Monitoring Based on Fog Computing Paradigm and Internet of Things , 2019, IEEE Access.

[37]  Xin Qin,et al.  FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare , 2019, IEEE Intelligent Systems.

[38]  Haoran Yu,et al.  Visual Inspection with Federated Learning , 2019, ICIAR.

[39]  Gaurav Kapoor,et al.  Protection Against Reconstruction and Its Applications in Private Federated Learning , 2018, ArXiv.

[40]  Daguang Xu,et al.  Privacy-preserving Federated Brain Tumour Segmentation , 2019, MLMI@MICCAI.

[41]  Xiaoyan Sun,et al.  Model and Feature Aggregation Based Federated Learning for Multi-sensor Time Series Trend Following , 2019, IWANN.

[42]  Dhruv Garg,et al.  Performance Analysis of Distributed and Federated Learning Models on Private Data , 2019, Procedia Computer Science.

[43]  Cyril Allauzen,et al.  Federated Learning of N-Gram Language Models , 2019, CoNLL.

[44]  Andrew M. Dai,et al.  Federated and Differentially Private Learning for Electronic Health Records , 2019, ArXiv.

[45]  Jonas Geiping,et al.  Inverting Gradients - How easy is it to break privacy in federated learning? , 2020, NeurIPS.

[46]  Fei Chen,et al.  Federated Meta-Learning with Fast Convergence and Efficient Communication , 2018 .

[47]  Nicha C. Dvornek,et al.  Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results , 2020, Medical Image Anal..

[48]  Bing Ren,et al.  Parallel Distributed Logistic Regression for Vertical Federated Learning without Third-Party Coordinator , 2019, ArXiv.

[49]  Tomás Horváth,et al.  Evolutionary Federated Learning on EEG-data , 2019, ITAT.

[50]  S. Chen,et al.  FL-QSAR: a federated learning based QSAR prototype for collaborative drug discovery , 2020, bioRxiv.

[51]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[52]  Han Yu,et al.  FOCUS: Dealing with Label Quality Disparity in Federated Learning , 2020, Federated Learning.

[53]  Haoyi Xiong,et al.  SecureGBM: Secure Multi-Party Gradient Boosting , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[54]  Abdullatif Albaseer,et al.  Exploiting Unlabeled Data in Smart Cities using Federated Learning , 2020, ArXiv.

[55]  Ying-Chang Liang,et al.  Federated Learning in Mobile Edge Networks: A Comprehensive Survey , 2020, IEEE Communications Surveys & Tutorials.

[56]  Yu Tian,et al.  A multicenter random forest model for effective prognosis prediction in collaborative clinical research network , 2020, Artif. Intell. Medicine.

[57]  Bing Chen,et al.  PEFL: A Privacy-Enhanced Federated Learning Scheme for Big Data Analytics , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[58]  Rui Li,et al.  Online Federated Multitask Learning , 2020 .

[59]  Eunho Yang,et al.  Federated Continual Learning with Adaptive Parameter Communication , 2020, ArXiv.

[60]  Yang Liu,et al.  Secure Federated Transfer Learning , 2018, ArXiv.

[61]  Osvaldo Simeone,et al.  Federated Neuromorphic Learning of Spiking Neural Networks for Low-Power Edge Intelligence , 2019, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[62]  Tianjian Chen,et al.  HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalography , 2019, ArXiv.

[63]  Kejiang Ye,et al.  FFD: A Federated Learning Based Method for Credit Card Fraud Detection , 2019, BigData.

[64]  Abdullatif Albaseer,et al.  Federated Learning for Localization: A Privacy-Preserving Crowdsourcing Method , 2020, ArXiv.

[65]  Li Huang,et al.  Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records , 2019, J. Biomed. Informatics.

[66]  Kin K. Leung,et al.  Data Selection for Federated Learning with Relevant and Irrelevant Data at Clients , 2020, ArXiv.

[67]  Swaroop Ramaswamy,et al.  Federated Learning for Emoji Prediction in a Mobile Keyboard , 2019, ArXiv.

[68]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[69]  Shu-Tao Xia,et al.  A Practical Privacy-preserving Method in Federated Deep Learning. , 2020 .

[70]  Thomas Brunschwiler,et al.  Privacy is What We Care About: Experimental Investigation of Federated Learning on Edge Devices , 2019, ArXiv.

[71]  Tianjian Chen,et al.  FedVision: An Online Visual Object Detection Platform Powered by Federated Learning , 2020, AAAI.

[72]  Spyridon Bakas,et al.  Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation , 2018, BrainLes@MICCAI.

[73]  Hamed Haddadi,et al.  Efficient and Private Federated Learning using TEE , 2019 .

[74]  Choong Seon Hong,et al.  Blockchain-based Node-aware Dynamic Weighting Methods for Improving Federated Learning Performance , 2019, 2019 20th Asia-Pacific Network Operations and Management Symposium (APNOMS).

[75]  Zhu Han,et al.  Federated Learning for Edge Networks: Resource Optimization and Incentive Mechanism , 2019, IEEE Communications Magazine.

[76]  Yuanming Shi,et al.  A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression , 2019, ArXiv.

[77]  Samuel Marchal,et al.  DÏoT: A Federated Self-learning Anomaly Detection System for IoT , 2018, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

[78]  Ming Liu,et al.  Federated Transfer Reinforcement Learning for Autonomous Driving , 2019, ArXiv.

[79]  Peng Xiao,et al.  Averaging Is Probably Not the Optimum Way of Aggregating Parameters in Federated Learning , 2020, Entropy.

[80]  Qian Xu,et al.  Federated Topic Modeling , 2019, CIKM.

[81]  Victor C. M. Leung,et al.  Secure Distributed On-Device Learning Networks with Byzantine Adversaries , 2019, IEEE Network.

[82]  Tommaso Di Noia,et al.  Towards Effective Device-Aware Federated Learning , 2019, AI*IA.

[83]  Dong In Kim,et al.  Toward an Automated Auction Framework for Wireless Federated Learning Services Market , 2019, ArXiv.

[84]  Xing Xie,et al.  FedRec: Privacy-Preserving News Recommendation with Federated Learning , 2020, 2003.09592.

[85]  Miao Pan,et al.  Federated Learning in Vehicular Edge Computing: A Selective Model Aggregation Approach , 2020, IEEE Access.

[86]  Ming Liu,et al.  Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems , 2019, IEEE Robotics and Automation Letters.

[87]  Lalana Kagal,et al.  PrivacyFL: A Simulator for Privacy-Preserving and Secure Federated Learning , 2020, CIKM.

[88]  Han Yu,et al.  FedCoin: A Peer-to-Peer Payment System for Federated Learning , 2020, Federated Learning.

[89]  Kenneth D. Mandl,et al.  FADL: Federated-Autonomous Deep Learning for Distributed Electronic Health Record , 2018, ArXiv.

[90]  Deze Zeng,et al.  A Learning-Based Incentive Mechanism for Federated Learning , 2020, IEEE Internet of Things Journal.

[91]  Chen-Yu Wei,et al.  Federated Residual Learning , 2020, ArXiv.

[92]  Sophie Chabridon,et al.  Combining Federated and Active Learning for Communication-efficient Distributed Failure Prediction in Aeronautics , 2020, ArXiv.

[93]  Ronald M. Summers,et al.  The future of digital health with federated learning , 2020, npj Digital Medicine.

[94]  Kalikinkar Mandal,et al.  PrivFL: Practical Privacy-preserving Federated Regressions on High-dimensional Data over Mobile Networks , 2019, IACR Cryptol. ePrint Arch..

[95]  Han Yu,et al.  Privacy-preserving Heterogeneous Federated Transfer Learning , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[96]  Hao Deng,et al.  LoAdaBoost: Loss-Based AdaBoost Federated Machine Learning on medical Data , 2018, ArXiv.

[97]  Vladimir Vlassov,et al.  Human Activity Recognition Using Federated Learning , 2018, 2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom).

[98]  Xing Xie,et al.  FedNER: Medical Named Entity Recognition with Federated Learning , 2020 .

[99]  Runhua Xu,et al.  HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning , 2019, AISec@CCS.

[100]  Choong Seon Hong,et al.  Federated Learning Based Mobile Edge Computing for Augmented Reality Applications , 2020, 2020 International Conference on Computing, Networking and Communications (ICNC).

[101]  Yasaman Khazaeni,et al.  Bayesian Nonparametric Federated Learning of Neural Networks , 2019, ICML.

[102]  Symeon Chatzinotas,et al.  Federated Learning Meets Contract Theory: Energy-Efficient Framework for Electric Vehicle Networks , 2020, ArXiv.

[103]  Youn-Hee Han,et al.  Federated Reinforcement Learning for Training Control Policies on Multiple IoT Devices , 2020, Sensors.

[104]  Richard Nock,et al.  Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption , 2017, ArXiv.

[105]  Moming Duan,et al.  Astraea: Self-Balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications , 2019, 2019 IEEE 37th International Conference on Computer Design (ICCD).

[106]  Yong Zhang,et al.  Facing small and biased data dilemma in drug discovery with federated learning , 2020, bioRxiv.

[107]  Xiaoyan Sun,et al.  Federated Learning Assisted Interactive EDA with Dual Probabilistic Models for Personalized Search , 2019, ICSI.

[108]  Yuliang Shi,et al.  Power Demand Response Incentive Pricing Model , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[109]  Kesheng Wu,et al.  Federated Wireless Network Intrusion Detection , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[110]  Abdulrahman Alotaibi.  Wisdom of the machines : federated learning using OPAL , 2018 .

[111]  Jun Li,et al.  Performance Analysis and Optimization in Privacy-Preserving Federated Learning , 2020, ArXiv.

[112]  Dmitriy Dligach,et al.  Two-stage Federated Phenotyping and Patient Representation Learning , 2019, BioNLP@ACL.

[113]  Wojciech Samek,et al.  Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[114]  H. Vincent Poor,et al.  Federated Learning for Task and Resource Allocation in Wireless High-Altitude Balloon Networks , 2020, IEEE Internet of Things Journal.

[115]  Yasaman Khazaeni,et al.  Federated Learning with Matched Averaging , 2020, ICLR.

[116]  Cheng-Zhong Xu,et al.  FC-SLAM: Federated Learning Enhanced Distributed Visual-LiDAR SLAM In Cloud Robotic System , 2019, 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[117]  Jiawen Kang,et al.  Privacy-Preserving Traffic Flow Prediction: A Federated Learning Approach , 2020, IEEE Internet of Things Journal.

[118]  Hiroki Matsutani,et al.  An On-Device Federated Learning Approach for Cooperative Anomaly Detection , 2020, ArXiv.

[119]  Bart Vanrumste,et al.  Privacy preserving pregnancy weight gain management: demo abstract , 2019, SenSys.

[120]  Mats Jirstrand,et al.  A Performance Evaluation of Federated Learning Algorithms , 2018, DIDL@Middleware.

[121]  Tao Xiang,et al.  A training-integrity privacy-preserving federated learning scheme with trusted execution environment , 2020, Inf. Sci..

[122]  Athina Markopoulou,et al.  A Federated Learning Approach for Mobile Packet Classification , 2019, ArXiv.

[123]  Weisong Shi,et al.  Collaborative Learning on the Edges: A Case Study on Connected Vehicles , 2019, HotEdge.

[124]  Haithum Elhadi,et al.  Federated Uncertainty-Aware Learning for Distributed Hospital EHR Data , 2019, ArXiv.

[125]  Eryk Dutkiewicz,et al.  Energy Demand Prediction with Federated Learning for Electric Vehicle Networks , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[126]  Shuyue Wei,et al.  Profit Allocation for Federated Learning , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[127]  Peter B. Walker,et al.  Federated Learning for Healthcare Informatics , 2019, Journal of Healthcare Informatics Research.

[128]  Aris Gkoulalas-Divanis,et al.  Differential Privacy-enabled Federated Learning for Sensitive Health Data , 2019, ArXiv.

[129]  Anit Kumar Sahu,et al.  On the Convergence of Federated Optimization in Heterogeneous Networks , 2018, ArXiv.

[130]  Hubert Eichner,et al.  Federated Learning for Mobile Keyboard Prediction , 2018, ArXiv.

[131]  Xian Wu,et al.  Federated Learning for Vision-and-Language Grounding Problems , 2020, AAAI.

[132]  Xiaowen Chu,et al.  FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MEC , 2020, ArXiv.

[133]  Wenchao Huang,et al.  FLChain: A Blockchain for Auditable Federated Learning with Trust and Incentive , 2019, 2019 5th International Conference on Big Data Computing and Communications (BIGCOM).

[134]  Walid Saad,et al.  Distributed Federated Learning for Ultra-Reliable Low-Latency Vehicular Communications , 2018, IEEE Transactions on Communications.

[135]  Omid Semiari,et al.  Federated Deep Learning for Immersive Virtual Reality over Wireless Networks , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[136]  Tianjian Chen,et al.  A Secure Federated Transfer Learning Framework , 2020, IEEE Intelligent Systems.

[137]  Tianjian Chen,et al.  Federated Machine Learning: Concept and Applications , 2019 .

[138]  Tassilo Klein,et al.  Differentially Private Federated Learning: A Client Level Perspective , 2017, ArXiv.

[139]  Yan Zhang,et al.  Differentially Private Asynchronous Federated Learning for Mobile Edge Computing in Urban Informatics , 2020, IEEE Transactions on Industrial Informatics.

[140]  Tianjian Chen,et al.  A Fairness-aware Incentive Scheme for Federated Learning , 2020, AIES.

[141]  Markus Fiedler,et al.  Privacy Preserving QoE Modeling using Collaborative Learning , 2019, Internet-QoE'19.

[142]  Emil Gustavsson,et al.  Federated Learning of Deep Neural Decision Forests , 2019, International Conference on Machine Learning, Optimization, and Data Science.