Distributed Primal-Dual Optimization for Online Multi-Task Learning

Conventional online multi-task learning algorithms suffer from two critical limitations: 1) Heavy communication caused by delivering high velocity of sequential data to a central machine; 2) Expensive runtime complexity for building task relatedness. To address these issues, in this paper we consider a setting where multiple tasks are geographically located in different places, where one task can synchronize data with others to leverage knowledge of related tasks. Specifically, we propose an adaptive primal-dual algorithm, which not only captures task-specific noise in adversarial learning but also carries out a projection-free update with runtime efficiency. Moreover, our model is well-suited to decentralized periodic-connected tasks as it allows the energy-starved or bandwidth-constraint tasks to postpone the update. Theoretical results demonstrate the convergence guarantee of our distributed algorithm with an optimal regret. Empirical results confirm that the proposed model is highly effective on various real-world datasets.

[1]  Patrick Gallinari,et al.  A distributed Frank–Wolfe framework for learning low-rank matrices with the trace norm , 2018, Machine Learning.

[2]  Philip M. Long,et al.  Online Multitask Learning , 2006, COLT.

[3]  Jiayu Zhou,et al.  Efficient multi-task feature learning with calibration , 2014, KDD.

[4]  Bernt Schiele,et al.  Scalable Multitask Representation Learning for Scene Classification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Javier Matamoros.  Asynchronous online ADMM for consensus problems , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Sinno Jialin Pan,et al.  Distributed Multi-Task Relationship Learning , 2017, KDD.

[7]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[8]  Bu-Sung Lee,et al.  Distributed multi-task classification: a decentralized online learning approach , 2018, Machine Learning.

[9]  Ambuj Tewari,et al.  Composite objective mirror descent , 2010, COLT 2010.

[10]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[11]  Ameet Talwalkar,et al.  Federated Multi-Task Learning , 2017, NIPS.

[12]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[13]  Peng Yang,et al.  Robust Online Multi-Task Learning with Correlative and Personalized Structures , 2017, IEEE Transactions on Knowledge and Data Engineering.

[14]  Jiayu Zhou,et al.  Modeling disease progression via multi-task learning , 2013, NeuroImage.

[15]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[16]  Avishek Saha,et al.  Online Learning of Multiple Tasks and Their Relationships , 2011, AISTATS.

[17]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[18]  Yiming Yang,et al.  Adaptive Smoothed Online Multi-Task Learning , 2016, NIPS.

[19]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[21]  Heinz H. Bauschke,et al.  Fixed-Point Algorithms for Inverse Problems in Science and Engineering , 2011, Springer Optimization and Its Applications.

[22]  Mladen Kolar,et al.  Distributed Multi-Task Learning with Shared Representation , 2016, ArXiv.

[23]  Tong Zhang,et al.  Projection-free Distributed Online Learning in Networks , 2017, ICML.

[24]  Jiayu Zhou,et al.  Privacy-Preserving Distributed Multi-Task Learning with Asynchronous Updates , 2017, KDD.

[25]  Jiayu Zhou,et al.  Confidence Weighted Multitask Learning , 2019, AAAI.

[26]  Tianbao Yang,et al.  SVD-free Convex-Concave Approaches for Nuclear Norm Regularization , 2017, IJCAI.

[27]  Jasha Droppo,et al.  Multi-task learning in deep neural networks for improved phoneme recognition , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[28]  Edoardo Amaldi,et al.  On the Approximability of Minimizing Nonzero Variables or Unsatisfied Relations in Linear Systems , 1998, Theor. Comput. Sci..

[29]  Lie Wang,et al.  Multivariate Regression with Calibration , 2013, NIPS.

[30]  Jiayu Zhou,et al.  Asynchronous Multi-task Learning , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[31]  Elad Hazan,et al.  Projection-free Online Learning , 2012, ICML.

[32]  Ramesh C. Jain,et al.  Collaborative online learning of user generated content , 2011, CIKM '11.

[33]  Shai Shalev-Shwartz,et al.  Online Learning and Online Convex Optimization , 2012, Found. Trends Mach. Learn..