Bayesian LASSO in a distributed architecture

We present a distributed framework for finding the full posterior distribution associated with LASSO problems. We leverage our recent results of formulating Bayesian inference as a KL divergence minimization problem that can be solved with linear algebra updates and a series of convex point estimation problems. We show that drawing samples from the Bayesian LASSO posterior can be done by iteratively solving LASSO problems in parallel. Motivated by wearable applications where (a) the energy cost of continuous wireless transmission is prohibitive and (b) cloud storage of data induces privacy vulnerabilities, we propose a class of `analog-to-information' architectures that only transmit the minimal relevant information (e.g. the posterior) for optimal decision-making. We instantiate this result with an analog-implementable solver and show that the posterior can be calculated with systems of low-energy analog circuits in a distributed manner.

[1]  Rui Ma,et al.  Efficient Bayesian inference methods via convex optimization and optimal transport , 2013, 2013 IEEE International Symposium on Information Theory.

[2]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[3]  Christopher J. Rozell,et al.  Low Power Sparse Approximation on Reconfigurable Analog Hardware , 2012, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[4]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[5]  Peter Dittrich,et al.  Understanding Networks of Computing Chemical Droplet Neurons Based on Information Flow , 2015, Int. J. Neural Syst..

[6]  Todd P. Coleman,et al.  Dynamic and Succinct Statistical Analysis of Neuroscience Data , 2014, Proceedings of the IEEE.

[7]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[8]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[9]  Todd P. Coleman,et al.  A scalable framework to transform samples from one continuous distribution to another , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[10]  Jeffrey M. Hausdorff,et al.  Physionet: Components of a New Research Resource for Complex Physiologic Signals". Circu-lation Vol , 2000 .

[11]  A. Ng Feature selection, L1 vs. L2 regularization, and rotational invariance , 2004, Twenty-first international conference on Machine learning - ICML '04.

[12]  2015 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2015, Orlando, FL, USA, December 14-16, 2015 , 2015, IEEE Global Conference on Signal and Information Processing.

[13]  J. Berger Statistical Decision Theory and Bayesian Analysis , 1988 .

[14]  Richard G. Baraniuk,et al.  Sparse Coding via Thresholding and Local Competition in Neural Circuits , 2008, Neural Computation.

[15]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[16]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[17]  Carla E. Brodley,et al.  Proceedings of the twenty-first international conference on Machine learning , 2004, International Conference on Machine Learning.

[18]  Tsachy Weissman,et al.  Justification of logarithmic loss via the benefit of side information , 2014, ISIT.

[19]  Youssef M. Marzouk,et al.  Bayesian inference with optimal maps , 2011, J. Comput. Phys..

[20]  Richard G. Baraniuk,et al.  A Field Guide to Forward-Backward Splitting with a FASTA Implementation , 2014, ArXiv.

[21]  Demba Ba,et al.  Robust spectrotemporal decomposition by iteratively reweighted least squares , 2014, Proceedings of the National Academy of Sciences.

[22]  G. Casella,et al.  The Bayesian Lasso , 2008 .

[23]  Raeed H. Chowdhury,et al.  Epidermal Electronics , 2011, Science.

[24]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[25]  Antonio Ortega,et al.  Signal compression in wireless sensor networks , 2012, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[26]  O. Ernst,et al.  ON THE CONVERGENCE OF GENERALIZED POLYNOMIAL CHAOS EXPANSIONS , 2011 .

[27]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .