A new class of distributed optimization algorithms: application to regression of distributed data

In a distributed optimization problem, the complete problem information is not available at a single location but is rather distributed among different agents in a multi-agent system. In the problems studied in the literature, each agent has an objective function and the network goal is to minimize the sum of the agents’ objective functions over a constraint set that is globally known. In this paper, we study a generalization of the above distributed optimization problem. In particular, the network objective is to minimize a function of the sum of the individual objective functions over the constraint set. The ‘outer’ function and the constraint set are known to all the agents. We discuss an algorithm and prove its convergence, and then discuss extensions to more general and complex distributed optimization problems. We provide a motivation for our algorithms through the example of distributed regression of distributed data.

[1]  H. Vincent Poor,et al.  A Collaborative Training Algorithm for Distributed Learning , 2009, IEEE Transactions on Information Theory.

[2]  Angelia Nedic,et al.  Distributed Non-Autonomous Power Control through Distributed Convex Optimization , 2009, IEEE INFOCOM 2009.

[3]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[4]  John N. Tsitsiklis,et al.  Distributed subgradient methods and quantization effects , 2008, 2008 47th IEEE Conference on Decision and Control.

[5]  Ee-Chien Chang,et al.  Distributed multivariate regression based on influential observations , 2003, KDD '03.

[6]  Angelia Nedic,et al.  Incremental Stochastic Subgradient Algorithms for Convex Optimization , 2008, SIAM J. Optim..

[7]  Soummya Kar,et al.  Distributed Algorithms in Sensor Networks , 2010 .

[8]  Xiaodong Lin,et al.  Secure Regression on Distributed Databases , 2005 .

[9]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[10]  Paul R. Cohen,et al.  Very Predictive Ngrams for Space-Limited Probabilistic Models , 2003, IDA.

[11]  Hillol Kargupta,et al.  A Scalable Local Algorithm for Distributed Multivariate Regression , 2008, Stat. Anal. Data Min..

[12]  Hillol Kargupta,et al.  Distributed Multivariate Regression Using Wavelet-Based Collective Data Mining , 2001, J. Parallel Distributed Comput..

[13]  Angelia Nedic,et al.  Asynchronous gossip algorithms for stochastic optimization , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[14]  R. Olfati-Saber,et al.  Consensus Filters for Sensor Networks and Distributed Sensor Fusion , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[15]  Yan Xing,et al.  Distributed Regression for Heterogeneous Data Sets , 2003, IDA.

[16]  Angelia Nedic,et al.  Distributed Stochastic Subgradient Projection Algorithms for Convex Optimization , 2008, J. Optim. Theory Appl..

[17]  Xiaodong Lin,et al.  Privacy preserving regression modelling via distributed computation , 2004, KDD.

[18]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[19]  Christian Borgelt,et al.  Advances in Intelligent Data Analysis V , 2003, Lecture Notes in Computer Science.

[20]  H. Vincent Poor,et al.  Regression in sensor networks: training distributively with alternating projections , 2005, SPIE Optics + Photonics.

[21]  C. Guestrin,et al.  Distributed regression: an efficient framework for modeling sensor network data , 2004, Third International Symposium on Information Processing in Sensor Networks, 2004. IPSN 2004.

[22]  V. Delouille,et al.  Robust distributed estimation in sensor networks using the embedded polygons algorithm , 2004, Third International Symposium on Information Processing in Sensor Networks, 2004. IPSN 2004.