Manifold Gradient Descent Solves Multi-Channel Sparse Blind Deconvolution Provably and Efficiently

Multi-channel sparse blind deconvolution refers to the problem of learning an unknown filter by observing its circulant convolutions with multiple input signals that are sparse. It is challenging to learn the filter efficiently due to the bilinear structure of the observations with respect to the unknown filter and inputs, leading to global ambiguities of identification. We propose a novel approach based on nonconvex optimization over the sphere manifold by minimizing a smooth surrogate of the sparsity-promoting loss function. It is demonstrated that manifold gradient descent with random initializations provably recovers the filter, up to scaling and shift ambiguities, as soon as the number of observations is sufficiently large under a suitable random data model. Numerical experiments are conducted to illustrate the efficiency of the proposed method with comparisons to existing methods.

[1]  Yudong Chen,et al.  Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimation: Recent Theory and Fast Algorithms via Convex and Nonconvex Optimization , 2018, IEEE Signal Processing Magazine.

[2]  Martin J. Wainwright,et al.  High-Dimensional Statistics , 2019 .

[3]  Xiaodong Li,et al.  Phase Retrieval via Wirtinger Flow: Theory and Algorithms , 2014, IEEE Transactions on Information Theory.

[4]  Max Simchowitz,et al.  Low-rank Solutions of Linear Matrix Equations via Procrustes Flow , 2015, ICML.

[5]  Yanjun Li,et al.  Global Geometry of Multichannel Sparse Blind Deconvolution on the Sphere , 2018, NeurIPS.

[6]  Xiaodong Li,et al.  Rapid, Robust, and Reliable Blind Deconvolution via Nonconvex Optimization , 2016, Applied and Computational Harmonic Analysis.

[7]  Yuxin Chen,et al.  Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval and Matrix Completion , 2018, ICML.

[8]  Yuejie Chi,et al.  Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent , 2020, J. Mach. Learn. Res..

[9]  Yuxin Chen,et al.  Nonconvex Matrix Factorization from Rank-One Measurements , 2019, AISTATS.

[10]  John Wright,et al.  A Geometric Analysis of Phase Retrieval , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[11]  Tengyu Ma,et al.  Matrix Completion has No Spurious Local Minimum , 2016, NIPS.

[12]  Eero P. Simoncelli,et al.  Recovery of Sparse Translation-Invariant Signals With Continuous Basis Pursuit , 2011, IEEE Transactions on Signal Processing.

[13]  Yanjun Li,et al.  Blind Gain and Phase Calibration via Sparse Spectral Methods , 2017, IEEE Transactions on Information Theory.

[14]  Justin Romberg,et al.  Multichannel myopic deconvolution in underwater acoustic channels via low-rank recovery. , 2017, The Journal of the Acoustical Society of America.

[15]  John Wright,et al.  Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture , 2015, IEEE Transactions on Information Theory.

[16]  Seungyong Lee,et al.  Fast motion deblurring , 2009, ACM Trans. Graph..

[17]  Liming Wang,et al.  Blind Deconvolution From Multiple Sparse Inputs , 2016, IEEE Signal Processing Letters.

[18]  Nicholas J. Higham,et al.  Functions of matrices - theory and computation , 2008 .

[19]  Yuejie Chi,et al.  A Non-Convex Approach To Joint Sensor Calibration And Spectrum Estimation , 2018, 2018 IEEE Statistical Signal Processing Workshop (SSP).

[20]  Yuejie Chi,et al.  Guaranteed Blind Sparse Spikes Deconvolution via Lifting and Convex Optimization , 2015, IEEE Journal of Selected Topics in Signal Processing.

[21]  Zhihui Zhu,et al.  A Nonconvex Approach for Exact and Efficient Multichannel Sparse Blind Deconvolution , 2019, NeurIPS.

[22]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..

[23]  Yu Bai,et al.  Subgradient Descent Learns Orthogonal Dictionaries , 2018, ICLR.

[24]  Junfeng Yang,et al.  An Efficient TVL1 Algorithm for Deblurring Multichannel Images Corrupted by Impulsive Noise , 2009, SIAM J. Sci. Comput..

[25]  Nathan Srebro,et al.  Global Optimality of Local Search for Low Rank Matrix Recovery , 2016, NIPS.

[26]  Sanjeev Arora,et al.  New Algorithms for Learning Incoherent and Overcomplete Dictionaries , 2013, COLT.

[27]  Y-Lan Boureau,et al.  Learning Convolutional Feature Hierarchies for Visual Recognition , 2010, NIPS.

[28]  Yi Zheng,et al.  No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis , 2017, ICML.

[29]  Dmitriy Drusvyatskiy,et al.  Composite optimization for robust blind deconvolution , 2019, ArXiv.

[30]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[31]  Yuxin Chen,et al.  Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution , 2017, Found. Comput. Math..

[32]  S.C. Douglas,et al.  Multichannel blind deconvolution and equalization using the natural gradient , 1997, First IEEE Signal Processing Workshop on Signal Processing Advances in Wireless Communications.

[33]  David Steurer,et al.  Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method , 2014, STOC.

[34]  Prateek Jain,et al.  Learning Sparsely Used Overcomplete Dictionaries , 2014, COLT.

[35]  Yonina C. Eldar,et al.  Sensor Calibration for Off-the-Grid Spectral Estimation , 2017, Applied and Computational Harmonic Analysis.

[36]  Yingbin Liang,et al.  Provable Non-convex Phase Retrieval with Outliers: Median TruncatedWirtinger Flow , 2016, ICML.

[37]  Yuxin Chen,et al.  Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview , 2018, IEEE Transactions on Signal Processing.

[38]  Song Li,et al.  Identifiability of Multichannel Blind Deconvolution and Nonconvex Regularization Algorithm , 2018, IEEE Transactions on Signal Processing.

[39]  Rémi Gribonval,et al.  Convex Optimization Approaches for Blind Sensor Calibration Using Sparsity , 2013, IEEE Transactions on Signal Processing.

[40]  Xiaodong Li,et al.  Optimal Rates of Convergence for Noisy Sparse Phase Retrieval via Thresholded Wirtinger Flow , 2015, ArXiv.

[41]  Yoram Bresler,et al.  Learning Sparsifying Transforms , 2013, IEEE Transactions on Signal Processing.

[42]  Jean-Christophe Pesquet,et al.  Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed ${\ell _1}/{\ell _2}$ Regularization , 2014, IEEE Signal Processing Letters.

[43]  Zhihui Zhu,et al.  Geometric Analysis of Nonconvex Optimization Landscapes for Overcomplete Learning , 2020, ICLR.

[44]  Yuxin Chen,et al.  Gradient descent with random initialization: fast global convergence for nonconvex phase retrieval , 2018, Mathematical Programming.

[45]  Justin Romberg,et al.  Fast and Guaranteed Blind Multichannel Deconvolution Under a Bilinear System Model , 2016, IEEE Transactions on Information Theory.

[46]  Qing Qu,et al.  From Symmetry to Geometry: Tractable Nonconvex Problems , 2020, ArXiv.

[47]  Thomas Strohmer,et al.  Self-calibration and biconvex compressive sensing , 2015, ArXiv.

[48]  John Wright,et al.  Efficient Dictionary Learning with Gradient Descent , 2018, ICML.

[49]  Thomas Strohmer,et al.  Self-Calibration and Bilinear Inverse Problems via Linear Least Squares , 2016, SIAM J. Imaging Sci..

[50]  Karl J. Friston,et al.  Modeling regional and psychophysiologic interactions in fMRI: the importance of hemodynamic deconvolution , 2003, NeuroImage.

[51]  Yanning Zhang,et al.  Multi-Observation Blind Deconvolution with an Adaptive Sparse Prior , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Justin K. Romberg,et al.  Blind Deconvolution Using Convex Programming , 2012, IEEE Transactions on Information Theory.

[53]  Yuling Yan,et al.  Convex and Nonconvex Optimization Are Both Minimax-Optimal for Noisy Blind Deconvolution , 2020, ArXiv.

[54]  Sunav Choudhary,et al.  Sparse blind deconvolution: What cannot be done , 2014, 2014 IEEE International Symposium on Information Theory.

[55]  Jing Wang,et al.  Blind Deconvolution for Symmetric Point-spread Functions , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[56]  Kjetil F. Kaaresen,et al.  Multichannel blind deconvolution of seismic signals , 1998 .

[57]  Yingbin Liang,et al.  A Nonconvex Approach for Phase Retrieval: Reshaped Wirtinger Flow and Incremental Algorithms , 2017, J. Mach. Learn. Res..

[58]  Kyle Luh,et al.  Dictionary Learning With Few Samples and Matrix Concentration , 2015, IEEE Transactions on Information Theory.

[59]  Sundeep Prabhakar Chepuri,et al.  Blind Calibration of Sparse Arrays for DOA Estimation with Analog and One-bit Measurements , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[60]  Prateek Jain,et al.  Non-convex Robust PCA , 2014, NIPS.

[61]  Yanning Zhang,et al.  Multi-image Blind Deblurring Using a Coupled Adaptive Sparse Prior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[62]  Thomas Strohmer Four short stories about Toeplitz matrix calculations , 2000 .

[63]  고봉환,et al.  Minimum Entropy Deconvolution을 이용한 회전체 시스템의 결함 모니터링 , 2013 .

[64]  John Wright,et al.  Structured Local Optima in Sparse Blind Deconvolution , 2018, IEEE Transactions on Information Theory.

[65]  Anastasios Kyrillidis,et al.  Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach , 2016, AISTATS.

[66]  Rémi Gribonval,et al.  Blind calibration for compressed sensing by convex optimization , 2011, 1111.7248.

[67]  Yanjun Li,et al.  A Unified Framework for Identifiability Analysis in Bilinear Inverse Problems with Applications to Subspace and Sparsity Models , 2015, ArXiv.

[68]  Yuejie Chi,et al.  Self-Calibrated Super Resolution , 2019, 2019 53rd Asilomar Conference on Signals, Systems, and Computers.

[69]  Afonso S. Bandeira,et al.  On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimization , 2018, SIAM J. Optim..

[70]  Yuxin Chen,et al.  Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems , 2015, NIPS.

[71]  Yi Ma,et al.  Complete Dictionary Learning via 𝓁4-Norm Maximization over the Orthogonal Group , 2019, J. Mach. Learn. Res..

[72]  John Wright,et al.  Geometry and Symmetry in Short-and-Sparse Deconvolution , 2019, ICML.

[73]  Huan Wang,et al.  Exact Recovery of Sparsely-Used Dictionaries , 2012, COLT.

[74]  Zhihui Zhu,et al.  Global Optimality in Low-Rank Matrix Optimization , 2017, IEEE Transactions on Signal Processing.

[75]  Sanjeev Arora,et al.  Simple, Efficient, and Neural Algorithms for Sparse Coding , 2015, COLT.