Compressive System Identification (CSI): Theory and applications of exploiting sparsity in the analysis of high-dimensional dynamical systems

The information content of many phenomena of practical interest is often much less than what is suggested by their actual size. As an inspiring example, one active research area in biology is to understand the relations between the genes. While the number of genes in a so-called gene network can be large, the number of contributing genes to each given gene in the network is usually small compared to the size of the network. In other words, the behavior of each gene can be expressed as a sparse combination of other genes. The purpose of this thesis is to develop new theory and algorithms for exploiting this type of simplicity in the analysis of high-dimensional dynamical systems with a particular focus on system identification and estimation. In particular, we consider systems with a high-dimensional but sparse impulse response, large-scale interconnected dynamical systems when the associated graph has a sparse flow, linear time-varying systems with few piecewise-constant parameter changes, and systems with a high-dimensional but sparse initial state. We categorize all of these problems under the common theme of Compressive System Identification (CSI) in which one aims at identifying some facts (e.g., the impulse response of the system, the underlying topology of the interconnected graph, or the initial state of the system) about the system under study from the smallest possible number of observations. Our work is inspired by the field of Compressive Sensing (CS) which is a recent paradigm in signal processing for sparse signal recovery. The CS recovery problem states that a sparse signal can be recovered from a small number of random linear measurements. Compared to the standard CS setup, however, we deal with structured sparse signals (e.g., block-sparse signals) and structured measurement matrices (e.g., Toeplitz matrices) where the structure is implied by the system under study.

[1]  Michael B. Wakin,et al.  Concentration of Measure for Block Diagonal Matrices With Applications to Compressive Signal Processing , 2011, IEEE Transactions on Signal Processing.

[2]  Justin K. Romberg,et al.  Compressive Sensing by Random Convolution , 2009, SIAM J. Imaging Sci..

[3]  Michael B. Wakin,et al.  The geometry of low-dimensional signal models , 2007 .

[4]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[5]  Christopher J. Rozell,et al.  On the Relation between Block Diagonal Matrices and Compressive Toeplitz Matrices , 2011 .

[6]  Holger Rauhut,et al.  Compressive Sensing with structured random matrices , 2012 .

[7]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[8]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[9]  D. Donoho,et al.  Counting faces of randomly-projected polytopes when the projection radically lowers dimension , 2006, math/0607364.

[10]  Suk-Geun Hwang,et al.  Cauchy's Interlace Theorem for Eigenvalues of Hermitian Matrices , 2004, Am. Math. Mon..

[11]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[12]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[13]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[14]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[15]  Michael B. Wakin,et al.  The Restricted Isometry Property for Random Block Diagonal Matrices , 2012, ArXiv.

[16]  Holger Rauhut,et al.  Circulant and Toeplitz matrices in compressed sensing , 2009, ArXiv.

[17]  Emmanuel J. Candès,et al.  Quantitative Robust Uncertainty Principles and Optimally Sparse Decompositions , 2004, Found. Comput. Math..

[18]  David L. Donoho,et al.  High-Dimensional Centrally Symmetric Polytopes with Neighborliness Proportional to Dimension , 2006, Discret. Comput. Geom..

[19]  Yonina C. Eldar,et al.  Block sparsity and sampling over a union of subspaces , 2009, 2009 16th International Conference on Digital Signal Processing.

[20]  E.J. Candes.  Compressive Sampling , 2022 .

[21]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[22]  Tyrone L. Vincent,et al.  Concentration of Measure Inequalities for Toeplitz Matrices With Applications , 2011, IEEE Transactions on Signal Processing.

[23]  L. Breiman.  Better subset regression using the nonnegative garrote , 1995 .

[24]  Hamid D. Taghirad,et al.  Modeling and Parameter Identification of Harmonic Drive Systems , 1998 .

[25]  Jacob Roll,et al.  The use of nonnegative garrote for order selection of ARX models , 2008, 2008 47th IEEE Conference on Decision and Control.

[26]  Richard G. Baraniuk,et al.  Signal Processing With Compressive Measurements , 2010, IEEE Journal of Selected Topics in Signal Processing.

[27]  Sanjoy Dasgupta,et al.  An elementary proof of a theorem of Johnson and Lindenstrauss , 2003, Random Struct. Algorithms.

[28]  James E. Braun,et al.  An Inverse Gray-Box Model for Transient Building Load Prediction , 2002 .

[29]  Tyrone L. Vincent,et al.  Compressive System Identification of LTI and LTV ARX models , 2011, IEEE Conference on Decision and Control and European Control Conference.

[30]  M. Rudelson,et al.  Sparse reconstruction by convex relaxation: Fourier and Gaussian measurements , 2006, 2006 40th Annual Conference on Information Sciences and Systems.

[31]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[32]  A. Barabasi,et al.  Hierarchical Organization of Modularity in Metabolic Networks , 2002, Science.

[33]  Christopher J. Rozell,et al.  The Restricted Isometry Property for Echo State Networks with applications to sequence memory capacity , 2012, 2012 IEEE Statistical Signal Processing Workshop (SSP).

[34]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[35]  Michael B. Wakin,et al.  Concentration of measure for block diagonal measurement matrices , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[36]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[37]  Robert D. Nowak,et al.  Causal Network Inference Via Group Sparse Regularization , 2011, IEEE Transactions on Signal Processing.

[38]  Donatello Materassi,et al.  A modeling approach to multivariate analysis and clusterization theory , 2008 .

[39]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[40]  M. Rudelson,et al.  On sparse reconstruction from Fourier and Gaussian measurements , 2008 .

[41]  Tyrone L. Vincent,et al.  A Review of Sufficient Conditions for Structure Identification in Interconnected Systems , 2012 .

[42]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[43]  J. Romberg,et al.  Restricted Isometries for Partial Random Circulant Matrices , 2010, arXiv.org.

[44]  M. Ledoux.  The concentration of measure phenomenon , 2001 .

[45]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[46]  Stephen P. Boyd,et al.  Segmentation of ARX-models using sum-of-norms regularization , 2010, Autom..

[47]  Carlos Canudas de Wit,et al.  A survey of models, analysis tools and compensation methods for the control of machines with friction , 1994, Autom..

[48]  Tyrone L. Vincent,et al.  Concentration of measure inequalities for compressive toeplitz matrices with applications to detection and system identification , 2010, 49th IEEE Conference on Decision and Control (CDC).

[49]  Mark A. Davenport,et al.  Random Observations on Random Observations: Sparse Signal Acquisition and Processing , 2010 .

[50]  Donatello Materassi,et al.  Reduced complexity models in the identification of dynamical networks: Links with sparsification problems , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[51]  G. Lugosi,et al.  On Concentration-of-Measure Inequalities , 1998 .

[52]  Robert D. Nowak,et al.  Toeplitz Compressed Sensing Matrices With Applications to Sparse Channel Estimation , 2010, IEEE Transactions on Information Theory.

[53]  S. Mendelson,et al.  Uniform Uncertainty Principle for Bernoulli and Subgaussian Ensembles , 2006, math/0608665.

[54]  Yonina C. Eldar,et al.  Robust Recovery of Signals From a Structured Union of Subspaces , 2008, IEEE Transactions on Information Theory.

[55]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[56]  D. Donoho,et al.  Neighborliness of randomly projected simplices in high dimensions. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[57]  Emmanuel J. Candès,et al.  A Compressed Sensing Parameter Extraction Platform for Radar Pulse Signal Acquisition , 2012, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[58]  Magnus Egerstedt,et al.  Controllability of Multi-Agent Systems from a Graph-Theoretic Perspective , 2009, SIAM J. Control. Optim..

[59]  Tyrone L. Vincent,et al.  A tutorial on recovery conditions for compressive system identification of sparse channels , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[60]  T. Blumensath,et al.  Theory and Applications , 2011 .

[61]  Shinji Hara,et al.  Consensus in hierarchical multi-agent dynamical systems with low-rank interconnections: Analysis of stability and convergence rates , 2009, 2009 American Control Conference.

[62]  Yonina C. Eldar,et al.  Block-Sparse Signals: Uncertainty Relations and Efficient Recovery , 2009, IEEE Transactions on Signal Processing.

[63]  Tyrone L. Vincent,et al.  Compressive System Identification in the Linear Time-Invariant framework , 2011, IEEE Conference on Decision and Control and European Control Conference.

[64]  Justin K. Romberg,et al.  Beyond Nyquist: Efficient Sampling of Sparse Bandlimited Signals , 2009, IEEE Transactions on Information Theory.

[65]  Deanna Needell,et al.  Uniform Uncertainty Principle and Signal Recovery via Regularized Orthogonal Matching Pursuit , 2007, Found. Comput. Math..

[66]  Michael B. Wakin,et al.  Concentration of measure for block diagonal matrices with repeated blocks , 2010, 2010 44th Annual Conference on Information Sciences and Systems (CISS).

[67]  Stephen J. Wright,et al.  Toeplitz-Structured Compressed Sensing Matrices , 2007, 2007 IEEE/SP 14th Workshop on Statistical Signal Processing.

[68]  L. Carin,et al.  Compressive particle filtering for target tracking , 2009, 2009 IEEE/SP 15th Workshop on Statistical Signal Processing.

[69]  Holger Rauhut,et al.  Suprema of Chaos Processes and the Restricted Isometry Property , 2012, ArXiv.

[70]  T. Moon,et al.  Mathematical Methods and Algorithms for Signal Processing , 1999 .

[71]  M. Talagrand,et al.  Probability in Banach Spaces: Isoperimetry and Processes , 1991 .

[72]  Donatello Materassi,et al.  Topological identification in networks of dynamical systems , 2008, 2008 47th IEEE Conference on Decision and Control.

[73]  Tyrone L. Vincent,et al.  Observability With Random Observations , 2014, IEEE Transactions on Automatic Control.

[74]  M. Wakin,et al.  Exact topology identification of large-scale interconnected dynamical systems from compressive observations , 2011, Proceedings of the 2011 American Control Conference.

[75]  Richard G. Baraniuk,et al.  Random Filters for Compressive Sampling and Reconstruction , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[76]  Tyrone L. Vincent,et al.  On the observability of linear systems from random, compressive measurements , 2010, 49th IEEE Conference on Decision and Control (CDC).

[77]  Ting Sun,et al.  Single-pixel imaging via compressive sampling , 2008, IEEE Signal Process. Mag..

[78]  Massimo Fornasier,et al.  Compressive Sensing and Structured Random Matrices , 2010 .

[79]  R. Mantegna.  Hierarchical structure in financial markets , 1998, cond-mat/9802256.

[80]  Tyrone L. Vincent,et al.  Compressive topology identification of interconnected dynamic systems via Clustered Orthogonal Matching Pursuit , 2011, IEEE Conference on Decision and Control and European Control Conference.

[81]  Richard G. Baraniuk,et al.  Random Projections of Smooth Manifolds , 2009, Found. Comput. Math..

[82]  Dimitris Achlioptas,et al.  Database-friendly random projections: Johnson-Lindenstrauss with binary coins , 2003, J. Comput. Syst. Sci..

[83]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[84]  Michael B. Wakin,et al.  The Restricted Isometry Property for block diagonal matrices , 2011, 2011 45th Annual Conference on Information Sciences and Systems.

[85]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[86]  E. Candès.  The restricted isometry property and its implications for compressed sensing , 2008 .

[87]  D.J. Brady,et al.  Compression at the Physical Interface , 2008, IEEE Signal Processing Magazine.

[88]  Reza Olfati-Saber,et al.  Consensus and Cooperation in Networked Multi-Agent Systems , 2007, Proceedings of the IEEE.

[89]  Olgica Milenkovic,et al.  Subspace Pursuit for Compressive Sensing Signal Reconstruction , 2008, IEEE Transactions on Information Theory.

[90]  Emmanuel J. Candès,et al.  A Nonuniform Sampler for Wideband Spectrally-Sparse Environments , 2012, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[91]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[92]  S. Bjorklund,et al.  A review of time-delay estimation techniques , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[93]  Wei Dai,et al.  Observability of a Linear System Under Sparsity Constraints , 2012, IEEE Transactions on Automatic Control.

[94]  Robert D. Nowak,et al.  Compressed channel sensing , 2008, 2008 42nd Annual Conference on Information Sciences and Systems.

[95]  Chi-Tsong Chen,et al.  Linear System Theory and Design , 1995 .

[96]  Sami Karjalainen,et al.  Estimating static heat flows in buildings for energy allocation systems , 2006 .

[97]  R. DeVore,et al.  Instance-optimality in probability with an ℓ1-minimization decoder , 2009 .

[98]  Richard Baraniuk,et al.  Recovery of Clustered Sparse Signals from Compressive Measurements , 2009 .

[99]  Toshiharu Sugie,et al.  A new approach for modeling hybrid systems based on the minimization of parameter transition in linear time-varying models , 2010, 49th IEEE Conference on Decision and Control (CDC).