Model-distributed solution of regularized least-squares problem over sensor networks

We develop a fully-distributed iterative algorithm for finding a model-distributed least-squares solution of systems of linear equations over sensor networks. Here, model-distributed means the solution vector is distributed across the network rather than being replicated at each node. For this purpose, we devise a dual regularized least-squares problem via a suitable decomposition of the normal equations associated with the original problem. The resultant dual problem can be solved in a fully-decentralized and iterative manner by means of the diffusion-based Pareto optimization strategy. We verify the usefulness of the proposed algorithm via both theoretical analysis and numerical examples.

[1]  Ali H. Sayed,et al.  On the Learning Behavior of Adaptive Networks—Part I: Transient Analysis , 2013, IEEE Transactions on Information Theory.

[2]  Ali H. Sayed,et al.  Adaptive Networks , 2014, Proceedings of the IEEE.

[3]  Werner Kratz,et al.  A substitute of l’Hospital’s rule for matrices , 1987 .

[4]  Rosemary A. Renaut,et al.  A parallel multisplitting solution of the least squares problem , 1998, Numer. Linear Algebra Appl..

[5]  Kostas Berberidis,et al.  Distributed Diffusion-Based LMS for Node-Specific Adaptive Parameter Estimation , 2014, IEEE Transactions on Signal Processing.

[6]  Ali Sayed,et al.  Adaptation, Learning, and Optimization over Networks , 2014, Found. Trends Mach. Learn..

[7]  Ioannis D. Schizas,et al.  Distributed Recursive Least-Squares for Consensus-Based In-Network Adaptive Estimation , 2009, IEEE Transactions on Signal Processing.

[8]  Stephen P. Boyd,et al.  Fast linear iterations for distributed averaging , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[9]  Ali H. Sayed,et al.  Diffusion recursive least-squares for distributed estimation over adaptive networks , 2008, IEEE Transactions on Signal Processing.

[10]  Soummya Kar,et al.  Distributed Consensus Algorithms in Sensor Networks With Imperfect Communication: Link Failures and Channel Noise , 2007, IEEE Transactions on Signal Processing.

[11]  Ali H. Sayed,et al.  Diffusion strategies for adaptation and learning over networks: an examination of distributed strategies and network behavior , 2013, IEEE Signal Processing Magazine.

[12]  Stefan Werner,et al.  Reduced-complexity distributed least-squares estimation over adaptive networks , 2013, 2013 IEEE 14th Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[13]  Yih-Fang Huang,et al.  Adaptive Distributed Estimation Based on Recursive Least-Squares and Partial Diffusion , 2014, IEEE Transactions on Signal Processing.

[14]  Rosemary A. Renaut.  A Parallel Multisplitting Solution of the Least Squares Problem , 1998 .

[15]  Sergios Theodoridis,et al.  Adaptive Robust Distributed Learning in Diffusion Sensor Networks , 2011, IEEE Transactions on Signal Processing.

[16]  Carl Tim Kelley,et al.  Iterative methods for optimization , 1999, Frontiers in applied mathematics.

[17]  Ali H. Sayed,et al.  On the limiting behavior of distributed optimization strategies , 2012, 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[18]  R. Brualdi.  Combinatorial Matrix Classes , 2006 .

[19]  Ali H. Sayed,et al.  On the Learning Behavior of Adaptive Networks—Part II: Performance Analysis , 2013, IEEE Transactions on Information Theory.

[20]  Srdjan S. Stankovic,et al.  Decentralized Parameter Estimation by Consensus Based Stochastic Approximation , 2007, IEEE Transactions on Automatic Control.

[21]  Jingdong Chen,et al.  Acoustic MIMO Signal Processing , 2006 .

[22]  Yih-Fang Huang,et al.  Analysis of a reduced-communication diffusion LMS algorithm , 2014, Signal Process..

[23]  Yih-Fang Huang,et al.  Distributed Least Mean-Square Estimation With Partial Diffusion , 2014, IEEE Transactions on Signal Processing.

[24]  Ali H. Sayed,et al.  Distributed pareto-optimal solutions via diffusion adaptation , 2012, 2012 IEEE Statistical Signal Processing Workshop (SSP).

[25]  Ali H. Sayed,et al.  Diffusion Least-Mean Squares Over Adaptive Networks: Formulation and Performance Analysis , 2008, IEEE Transactions on Signal Processing.

[26]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[27]  Stefan Werner,et al.  Partial-diffusion recursive least-squares estimation over adaptive networks , 2013, 2013 5th IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[28]  Ali H. Sayed,et al.  Dictionary Learning Over Distributed Models , 2014, IEEE Transactions on Signal Processing.

[29]  Ali H. Sayed,et al.  Diffusion LMS Strategies for Distributed Estimation , 2010, IEEE Transactions on Signal Processing.

[30]  Wilfried N. Gansterer,et al.  Analysis and Comparison of Truly Distributed Solvers for Linear Least Squares Problems on Wireless Sensor Networks , 2014, Euro-Par.

[31]  D. O’Leary,et al.  Multi-Splittings of Matrices and Parallel Solution of Linear Systems , 1985 .

[32]  Ioannis D. Schizas,et al.  Distributed LMS for Consensus-Based In-Network Adaptive Processing , 2009, IEEE Transactions on Signal Processing.

[33]  Suleyman Serdar Kozat,et al.  Compressive Diffusion Strategies Over Distributed Networks for Reduced Communication Load , 2014, IEEE Transactions on Signal Processing.

[34]  Ali H. Sayed,et al.  Diffusion Adaptation over Networks , 2012, ArXiv.

[35]  Ali H. Sayed,et al.  Distributed Pareto Optimization via Diffusion Strategies , 2012, IEEE Journal of Selected Topics in Signal Processing.