System identification under non-negativity constraints - Applications in adaptive filtering and hyperspectral image analysis. (Indentification de système sous la contrainte de non-négativité - Applications dans filtrage adaptatif et analyse dimage hyperspectrale)

In many real-life phenomena due to the inherent physical characteristics of systems under investigation, non-negativity is a desired constraint that can be imposed on the parameters to estimate. The objective of this thesis is to investigate theories and algorithms for system identification under side constraints, in particular the non-negativity constraint and sum-to-one constraint over the vector of parameters to estimate. The first part of the thesis addresses the problem of online system identification subject to non-negativity constraints. The Non-negative Least-Mean-Square algorithm (NNLMS) and its variants are proposed. The stochastic behavior of these algorithms in non-stationnary environments is analytically studied. Finally, coupling these algorithms with optimization allows us to derive an LMS-type algorithm with L1-norm regularization The second part of the thesis focuses on a specific system identifcation problem - nonlinear spectral unmixing, with non-negativity and sum-to-one constraints. We formulate a new kernel-based paradigm that relies on the assumption that the mixing mechanism can be described by a linear mixture of endmember spectra, with additive nonlinear fluctuations defined in a reproducing kernel Hilbert space. A kernel-based algortihm, based on muli-kernel learning, is proposed to determine the fractional abundances of pure materials subject ot constraints. Finally, the spatila correlation between spectral signatures of neighboring pixels is used as prior infromation to improve the performance.

[1]  S. Haykin Adaptive Filters , 2007 .

[2]  Antonio Plaza,et al.  Recent Developments in Endmember Extraction and Spectral Unmixing , 2011 .

[3]  Zhigang Luo,et al.  Manifold Regularized Discriminative Nonnegative Matrix Factorization With Fast Gradient Descent , 2011, IEEE Transactions on Image Processing.

[4]  Lennart Ljung,et al.  Perspectives on system identification , 2010, Annu. Rev. Control..

[5]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[6]  H. O. Posten Multidimensional Gaussian Distributions , 1964 .

[7]  Antonio J. Plaza,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 1 Spectral–Spatial Hyperspectral Image Segmentation Using S , 2022 .

[8]  J. Karhunen,et al.  Advances in Nonlinear Blind Source Separation , 2003 .

[9]  J. Mercer Functions of Positive and Negative Type, and their Connection with the Theory of Integral Equations , 1909 .

[10]  Derek Rogge,et al.  Integration of spatial–spectral information for the improved extraction of endmembers , 2007 .

[11]  Nello Cristianini,et al.  Learning the Kernel Matrix with Semidefinite Programming , 2002, J. Mach. Learn. Res..

[12]  Amit Banerjee,et al.  A comparison of kernel functions for intimate mixture models , 2009, 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing.

[13]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[14]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[15]  José Carlos M. Bermudez,et al.  An improved statistical analysis of the least mean fourth (LMF) adaptive algorithm , 2003, IEEE Trans. Signal Process..

[16]  Seung-Jean Kim,et al.  Hyperspectral Image Unmixing via Alternating Projected Subgradients , 2007, 2007 Conference Record of the Forty-First Asilomar Conference on Signals, Systems and Computers.

[17]  Donghui Chen,et al.  Nonnegativity constraints in numerical analysis , 2009, The Birth of Numerical Analysis.

[18]  Wotao Yin,et al.  Bregman Iterative Algorithms for (cid:2) 1 -Minimization with Applications to Compressed Sensing ∗ , 2008 .

[19]  Sen Jia,et al.  Spectral and Spatial Complexity-Based Hyperspectral Unmixing , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[21]  Alfred O. Hero,et al.  Sparse LMS for system identification , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[22]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[23]  M. Zibulevsky,et al.  Sequential Subspace Optimization Method for Large-Scale Unconstrained Problems , 2005 .

[24]  Rob Heylen,et al.  Non-Linear Spectral Unmixing by Geodesic Simplex Volume Maximization , 2011, IEEE Journal of Selected Topics in Signal Processing.

[25]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[26]  V. Umapathi Reddy,et al.  Fixed point error analysis of the normalized ladder algorithm , 1982, ICASSP.

[27]  J. Yorke,et al.  Chaos: An Introduction to Dynamical Systems , 1997 .

[28]  B. Gaines SYSTEM IDENTIFICATION, APPROXIMATION AND COMPLEXITY , 1977 .

[29]  Hairong Qi,et al.  Endmember Extraction From Highly Mixed Data Using Minimum Volume Constrained Nonnegative Matrix Factorization , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[30]  A. Hero,et al.  Regularized Least-Mean-Square Algorithms , 2010, 1012.5066.

[31]  Tor Arne Johansen,et al.  Constrained and regularized system identification , 1997 .

[32]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[33]  Jian-Feng Cai,et al.  Linearized Bregman iterations for compressed sensing , 2009, Math. Comput..

[34]  H Lantéri,et al.  A general method to devise maximum-likelihood signal restoration multiplicative algorithms with non-negativity constraints , 2001, Signal Process..

[35]  Lorenzo Bruzzone,et al.  Kernel-based methods for hyperspectral image classification , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Sergios Theodoridis,et al.  Adaptive Learning in a World of Projections , 2011, IEEE Signal Processing Magazine.

[37]  Deniz Erdogmus,et al.  Learning mappings in brain machine interfaces with echo state networks , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[38]  Li Ma,et al.  Local Manifold Learning-Based $k$ -Nearest-Neighbor for Hyperspectral Image Classification , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[39]  David Haussler,et al.  Convolution kernels on discrete structures , 1999 .

[40]  Antonio J. Plaza,et al.  Region-Based Spatial Preprocessing for Endmember Extraction and Spectral Unmixing , 2011, IEEE Geoscience and Remote Sensing Letters.

[41]  Jon Atli Benediktsson,et al.  On the decomposition of Mars hyperspectral data by ICA and Bayesian positive source separation , 2008, Neurocomputing.

[42]  Antonio J. Plaza,et al.  Total Variation Spatial Regularization for Sparse Hyperspectral Unmixing , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[43]  J. Boardman Automating spectral unmixing of AVIRIS data using convex geometry concepts , 1993 .

[44]  R. Tibshirani,et al.  PATHWISE COORDINATE OPTIMIZATION , 2007, 0708.1485.

[45]  N. Aronszajn Theory of Reproducing Kernels. , 1950 .

[46]  Thomas L. Ainsworth,et al.  Exploiting manifold geometry in hyperspectral imagery , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[47]  Alexander Shapiro,et al.  Optimization Problems with Perturbations: A Guided Tour , 1998, SIAM Rev..

[48]  D. Brie,et al.  Separation of Non-Negative Mixture of Non-Negative Sources Using a Bayesian Approach and MCMC Sampling , 2006, IEEE Transactions on Signal Processing.

[49]  Luciano Alparone,et al.  Signal-dependent noise modelling and estimation of new-generation imaging spectrometers , 2009, 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing.

[50]  Qian Du,et al.  Improving the quality of extracted endmembers , 2009, 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing.

[51]  Antonio Plaza,et al.  NONLINEAR NEURAL NETWORK MIXTURE MODELS FOR FRACTIONAL ABUNDANCE ESTIMATION IN AVIRIS HYPERSPECTRAL IMAGES , 2005 .

[52]  Rama Chellappa,et al.  Kernel fully constrained least squares abundance estimates , 2007, 2007 IEEE International Geoscience and Remote Sensing Symposium.

[53]  Heesung Kwon,et al.  Kernel orthogonal subspace projection for hyperspectral signal classification , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[54]  Marco Diani,et al.  Signal-Dependent Noise Modeling and Model Parameter Estimation in Hyperspectral Images , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[55]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[56]  Daniel D. Lee,et al.  Bayesian regularization and nonnegative deconvolution for room impulse response estimation , 2006, IEEE Transactions on Signal Processing.

[57]  Mario Winter,et al.  N-FINDR: an algorithm for fast autonomous spectral end-member determination in hyperspectral data , 1999, Optics & Photonics.

[58]  S. Hook,et al.  The ASTER spectral library version 2.0 , 2009 .

[59]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[60]  Toon van Waterschoot,et al.  Static field estimation using a wireless sensor network based on the finite element method , 2011, 2011 4th IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[61]  Antonio Plaza,et al.  Hyperspectral unmixing: geometrical, statistical, and sparse regression-based approaches , 2010, Remote Sensing.

[62]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[63]  J. Minkoff,et al.  Comment on the "Unnecessary assumption of statistical independence between reference signal and filter weights in feedforward adaptive systems" , 2001, IEEE Trans. Signal Process..

[64]  Jie Chen,et al.  Nonlinear Unmixing of Hyperspectral Data Based on a Linear-Mixture/Nonlinear-Fluctuation Model , 2013, IEEE Transactions on Signal Processing.

[65]  Alina Zare,et al.  Spatial-spectral unmixing using fuzzy local information , 2011, 2011 IEEE International Geoscience and Remote Sensing Symposium.

[66]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[67]  José M. Bioucas-Dias,et al.  Minimum Volume Simplex Analysis: A Fast Algorithm to Unmix Hyperspectral Data , 2008, IGARSS 2008 - 2008 IEEE International Geoscience and Remote Sensing Symposium.

[68]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[69]  Antonio J. Plaza,et al.  Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[70]  V. P. Pauca,et al.  Nonnegative matrix factorization for spectral data analysis , 2006 .

[71]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[72]  Eweda Eweda,et al.  Transient and tracking performance bounds of the sign-sign algorithm , 1999, IEEE Trans. Signal Process..

[73]  Yuantao Gu,et al.  $l_{0}$ Norm Constraint LMS Algorithm for Sparse System Identification , 2009, IEEE Signal Processing Letters.

[74]  Paul Honeine,et al.  Non-negative distributed regression for data inference in wireless sensor networks , 2010, 2010 Conference Record of the Forty Fourth Asilomar Conference on Signals, Systems and Computers.

[75]  Robert M. May,et al.  Simple mathematical models with very complicated dynamics , 1976, Nature.

[76]  Qian Du,et al.  Nonlinear Spectral Mixture Analysis for Hyperspectral Imagery in an Unknown Environment , 2010, IEEE Geoscience and Remote Sensing Letters.

[77]  Jean-Yves Tourneret,et al.  Bayesian Estimation of Linear Mixtures Using the Normal Compositional Model. Application to Hyperspectral Imagery , 2010, IEEE Transactions on Image Processing.

[78]  Qian Du,et al.  Foreword to the Special Issue on Spectral Unmixing of Remotely Sensed Data , 2011 .

[79]  Chong-Yung Chi,et al.  A convex analysis-based minimum-volume enclosing simplex algorithm for hyperspectral unmixing , 2009, IEEE Trans. Signal Process..

[80]  Antonio J. Plaza,et al.  Sparse Unmixing of Hyperspectral Data , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[81]  Robert Price,et al.  A useful theorem for nonlinear devices having Gaussian inputs , 1958, IRE Trans. Inf. Theory.

[82]  L. A. Zadeh,et al.  From Circuit Theory to System Theory , 1962, Proceedings of the IRE.

[83]  Johnathan M. Bardsley,et al.  THE STABILIZING PROPERTIES OF NONNEGATIVITY CONSTRAINTS IN LEAST-SQUARES IMAGE RECONSTRUCTION , 2007 .

[84]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[85]  Xiaojun Wu,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[86]  Sérgio J. M. de Almeida,et al.  A statistical analysis of the affine projection algorithm for unity step size and autoregressive inputs , 2005, IEEE Transactions on Circuits and Systems I: Regular Papers.

[87]  Paul Honeine,et al.  Geometric Unmixing of Large Hyperspectral Images: A Barycentric Coordinate Approach , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[88]  Y. Nesterov Gradient methods for minimizing composite objective function , 2007 .

[89]  R. Bro,et al.  A fast non‐negativity‐constrained least squares algorithm , 1997 .

[90]  José M. Bioucas-Dias,et al.  Unmixing hyperspectral intimate mixtures , 2010, Remote Sensing.

[91]  Arie Yeredor,et al.  On the Role of Constraints in System Identification , 2006 .

[92]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[93]  Antonio J. Plaza,et al.  Spatial Preprocessing for Endmember Extraction , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[94]  Robert D. Nowak,et al.  A bound optimization approach to wavelet-based image deconvolution , 2005, IEEE International Conference on Image Processing 2005.

[95]  Jean-Yves Tourneret,et al.  Nonlinear unmixing of hyperspectral images using a generalized bilinear model , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[96]  Chein-I Chang,et al.  A quantitative and comparative analysis of linear and nonlinear spectral mixture models using radial basis function neural networks , 2001, IEEE Trans. Geosci. Remote. Sens..

[97]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .

[98]  Jean-Yves Tourneret,et al.  Enhancing Hyperspectral Image Unmixing With Spatial Correlations , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[99]  Bhaskar D. Rao,et al.  Sparse channel estimation via matching pursuit with application to equalization , 2002, IEEE Trans. Commun..

[100]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[101]  B. Hapke Bidirectional reflectance spectroscopy: 1. Theory , 1981 .

[102]  C. Richard Johnson,et al.  Sign-sign LMS convergence with independent stochastic inputs , 1990, IEEE Trans. Inf. Theory.

[103]  Alfred O. Hero,et al.  Joint Bayesian Endmember Extraction and Linear Unmixing for Hyperspectral Imagery , 2009, IEEE Transactions on Signal Processing.

[104]  Jean-Yves Tourneret,et al.  Supervised nonlinear spectral unmixing using a polynomial post nonlinear model for hyperspectral imagery , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[105]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[106]  Antonio J. Plaza,et al.  Unmixing Prior to Supervised Classification of Remotely Sensed Hyperspectral Images , 2011, IEEE Geoscience and Remote Sensing Letters.

[107]  José M. Bioucas-Dias,et al.  Vertex component analysis: a fast algorithm to unmix hyperspectral data , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[108]  José M. Bioucas-Dias,et al.  Nonlinear mixture model for hyperspectral unmixing , 2009, Remote Sensing.

[109]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[110]  Mark D. Plumbley Algorithms for nonnegative independent component analysis , 2003, IEEE Trans. Neural Networks.

[111]  Joseph Meola,et al.  Modeling and estimation of signal-dependent noise in hyperspectral imagery. , 2011, Applied optics.

[112]  M. Bertero,et al.  Nonnegative least-squares image deblurring: improved gradient projection approaches , 2010 .

[113]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[114]  Shengli Zhou,et al.  Application of compressive sensing to sparse channel estimation , 2010, IEEE Communications Magazine.

[115]  Alfonso Fernández-Manso,et al.  Spectral unmixing , 2012 .

[116]  T. R. Gosnell,et al.  Digital Image Reconstruction: Deblurring and Denoising , 2005 .

[117]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[118]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[119]  T. Warner,et al.  Spatial autocorrelation analysis of hyperspectral imagery for feature selection , 1997 .

[120]  Alexandros G. Dimakis,et al.  Sparse Recovery of Nonnegative Signals With Minimal Expansion , 2011, IEEE Transactions on Signal Processing.

[121]  Stephen J. Wright,et al.  Sparse Reconstruction by Separable Approximation , 2008, IEEE Transactions on Signal Processing.

[122]  Daniel D. Lee,et al.  Statistical signal processing with nonnegativity constraints , 2003, INTERSPEECH.

[123]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[124]  T. W. Ray,et al.  Nonlinear Spectral Mixing in Desert Vegetation , 1996 .

[125]  Shin'ichi Koike Analysis of the sign-sign algorithm based on Gaussian distributed tap weights , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[126]  Konstantinos Koutroumbas,et al.  A Novel Hierarchical Bayesian Approach for Sparse Semisupervised Hyperspectral Unmixing , 2012, IEEE Transactions on Signal Processing.

[127]  Paul Honeine,et al.  Online Prediction of Time Series Data With Kernels , 2009, IEEE Transactions on Signal Processing.

[128]  Stanley Osher,et al.  L1 unmixing and its application to hyperspectral image enhancement , 2009, Defense + Commercial Sensing.

[129]  J. B. Rosen The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints , 1960 .

[130]  M. Gevers,et al.  A personal view of the development of system identification: A 30-year journey through an exciting field , 2006, IEEE Control Systems.

[131]  M. V. Van Benthem,et al.  Fast algorithm for the solution of large‐scale non‐negativity‐constrained least squares problems , 2004 .