Distributed Online Learning With Multiple Kernels

In the Internet-of-Things (IoT) systems, there are plenty of informative data provided by a massive number of IoT devices (e.g., sensors). Learning a function from such data is of great interest in machine learning tasks for IoT systems. Focusing on streaming (or sequential) data, we present a privacy-preserving distributed online learning framework with multiplekernels (named DOMKL). The proposed DOMKL is devised by leveraging the principles of an online alternating direction of multipliers (OADMM) and a distributed Hedge algorithm. We theoretically prove that DOMKL over T time slots can achieve an optimal sublinear regret, implying that every learned function achieves the performance of the best function in hindsight as in the state-of-the-art centralized online learning method. Moreover, it is ensured that the learned functions of any two neighboring learners have a negligible difference as T grows, i.e., the so-called consensus constraints hold. Via experimental tests with various real datasets, we verify the effectiveness of the proposed DOMKL on regression and time-series prediction tasks.

[1]  Anusha Lalitha,et al.  Fully Decentralized Federated Learning , 2018 .

[2]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[3]  E. Massera,et al.  On field calibration of an electronic nose for benzene estimation in an urban pollution monitoring scenario , 2008 .

[4]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[5]  Yücel Saygin,et al.  Privacy-Preserving Learning Analytics: Challenges and Techniques , 2017, IEEE Transactions on Learning Technologies.

[6]  Terence C. Mills,et al.  Time series techniques for economists , 1990 .

[7]  Paul Honeine,et al.  Online Prediction of Time Series Data With Kernels , 2009, IEEE Transactions on Signal Processing.

[8]  Georgios B. Giannakis,et al.  Nonparametric Basis Pursuit via Sparse Kernel-Based Learning: A Unifying View with Advances in Blind Methods , 2013, IEEE Signal Processing Magazine.

[9]  S. Kak.  Information, physics, and computation , 1996 .

[10]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[11]  Bereket Tanju,et al.  A Machine Learning Decision-Support System Improves the Internet of Things’ Smart Meter Operations , 2017, IEEE Internet of Things Journal.

[12]  Ruixuan Li,et al.  Privacy-Preserving and Residential Context-Aware Online Learning for IoT-Enabled Energy Saving With Big Data Support in Smart Home Environment , 2019, IEEE Internet of Things Journal.

[13]  Yang Gao,et al.  Online Selective Kernel-Based Temporal Difference Learning , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Armin Dekorsy,et al.  Distributed Adaptive Learning With Multiple Kernels in Diffusion Networks , 2018, IEEE Transactions on Signal Processing.

[15]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[16]  Doreid Ammar,et al.  Operationalizing Solar Energy Predictions for Sustainable, Autonomous IoT Device Management , 2020, IEEE Internet of Things Journal.

[17]  Jie Chen,et al.  Diffusion adaptation over networks with kernel least-mean-square , 2015, 2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[18]  Sajjad Hussain Shah,et al.  A survey: Internet of Things (IOT) technologies, applications and challenges , 2016, 2016 IEEE Smart Energy Grid Engineering (SEGE).

[19]  Gunnar Rätsch,et al.  Large Scale Multiple Kernel Learning , 2006, J. Mach. Learn. Res..

[20]  Huaguang Zhang,et al.  Consensus-Based Distributed Cooperative Learning From Closed-Loop Neural Control Systems , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Sergios Theodoridis,et al.  Online Distributed Learning Over Networks in RKH Spaces Using Random Fourier Features , 2017, IEEE Transactions on Signal Processing.

[22]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[23]  Paul Honeine,et al.  Online Prediction of Time Series Data With Kernels , 2009, IEEE Trans. Signal Process..

[24]  Hubert Eichner,et al.  Federated Learning for Mobile Keyboard Prediction , 2018, ArXiv.

[25]  Charles A. Micchelli,et al.  Learning the Kernel Function via Regularization , 2005, J. Mach. Learn. Res..

[26]  Arindam Banerjee,et al.  Online Alternating Direction Method (longer version) , 2013, ArXiv.

[27]  Jeongmin Chae,et al.  Active Learning With Multiple Kernels , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[29]  Mianxiong Dong,et al.  Learning IoT in Edge: Deep Learning for the Internet of Things with Edge Computing , 2018, IEEE Network.

[30]  Martin J. Wainwright,et al.  High-Dimensional Statistics , 2019 .

[31]  Sébastien Bubeck,et al.  Introduction to Online Optimization , 2011 .

[32]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[33]  Bin Li,et al.  Online multiple kernel regression , 2014, KDD.

[34]  Markus Wagner,et al.  A detailed comparison of meta-heuristic methods for optimising wave energy converter placements , 2018, GECCO.

[35]  François Kawala,et al.  Prédictions d'activité dans les réseaux sociaux en ligne , 2013 .

[36]  Cheolhee Yoo,et al.  Comparative Assessment of Various Machine Learning‐Based Bias Correction Methods for Numerical Weather Prediction Model Forecasts of Extreme Air Temperatures in Urban Areas , 2020, Earth and Space Science.

[37]  Georgios B. Giannakis,et al.  Random Feature-based Online Multi-kernel Learning in Environments with Unknown Dynamics , 2017, J. Mach. Learn. Res..

[38]  Suleyman Serdar Kozat,et al.  Sequential Nonlinear Learning for Distributed Multiagent Systems via Extreme Learning Machines , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[39]  K. Hamidieh.  A data-driven statistical model for predicting the critical temperature of a superconductor , 2018, Computational Materials Science.

[40]  Armin Dekorsy,et al.  Distributed nonlinear regression using in-network processing with multiple Gaussian kernels , 2017, 2017 IEEE 18th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[41]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[42]  Alexander J. Smola,et al.  Online learning with kernels , 2001, IEEE Transactions on Signal Processing.

[43]  Chiou-Shann Fuh,et al.  Multiple Kernel Learning for Dimensionality Reduction , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Sneha A. Dalvi,et al.  Internet of Things for Smart Cities , 2017 .

[45]  G. Wahba.  Spline models for observational data , 1990 .

[46]  Elad Hazan,et al.  Introduction to Online Convex Optimization , 2016, Found. Trends Optim..

[47]  Kuang-Ching Wang,et al.  Review of Internet of Things (IoT) in Electric Power and Energy Systems , 2018, IEEE Internet of Things Journal.

[48]  Klaus-Robert Müller,et al.  Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[49]  Geoffrey Ye Li,et al.  Toward Intelligent Vehicular Networks: A Machine Learning Framework , 2018, IEEE Internet of Things Journal.

[50]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.