Global testing against sparse alternatives in time-frequency analysis

In this paper, an over-sampled periodogram higher criticism (OPHC) test is proposed for the global detection of sparse periodic effects in a complex-valued time series. An explicit minimax detection boundary is established between the rareness and weakness of the complex sinusoids hidden in the series. The OPHC test is shown to be asymptotically powerful in the detectable region. Numerical simulations illustrate and verify the effectiveness of the proposed test. Furthermore, the periodogram over-sampled by $O(\log N)$ is proven universally optimal in global testing for periodicities under a mild minimum separation condition.

[1]  Jian Li,et al.  Higher criticism: $p$-values and criticism , 2014, 1411.1437.

[2]  A. Juditsky,et al.  On detecting harmonic oscillations , 2013, 1301.5328.

[3]  Lee H. Dicker,et al.  Variance estimation in high-dimensional linear models , 2014 .

[4]  Yonina C. Eldar,et al.  Channel estimation in UWB channels using compressed sensing , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Yihong Wu,et al.  Optimal Detection of Sparse Mixtures Against a Given Null Distribution , 2014, IEEE Transactions on Information Theory.

[6]  Yonina C. Eldar,et al.  Sub-Nyquist radar prototype: Hardware and algorithm , 2014, IEEE Transactions on Aerospace and Electronic Systems.

[7]  Marco F. Duarte,et al.  Spectral compressive sensing , 2013 .

[8]  Sofia C. Olhede,et al.  The Whittle Likelihood for Complex-Valued Time Series , 2013 .

[9]  Gongguo Tang,et al.  Near minimax line spectral estimation , 2013, 2013 47th Annual Conference on Information Sciences and Systems (CISS).

[10]  Gongguo Tang,et al.  Atomic Norm Denoising With Applications to Line Spectral Estimation , 2012, IEEE Transactions on Signal Processing.

[11]  Emmanuel J. Candès,et al.  Super-Resolution from Noisy Data , 2012, Journal of Fourier Analysis and Applications.

[12]  Qiang Fu,et al.  Compressed Sensing of Complex Sinusoids: An Approach Based on Dictionary Refinement , 2012, IEEE Transactions on Signal Processing.

[13]  Emmanuel J. Candès,et al.  Towards a Mathematical Theory of Super‐resolution , 2012, ArXiv.

[14]  Gitta Kutyniok,et al.  1 . 2 Sparsity : A Reasonable Assumption ? , 2012 .

[15]  Yonina C. Eldar,et al.  Compressed Beamforming in Ultrasound Imaging , 2012, IEEE Transactions on Signal Processing.

[16]  Wenjing Liao,et al.  Coherence Pattern-Guided Compressive Sensing with Unresolved Grids , 2011, SIAM J. Imaging Sci..

[17]  Yonina C. Eldar,et al.  Sampling at the rate of innovation: theory and applications , 2012, Compressed Sensing.

[18]  Jiashun Jin,et al.  Optimal detection of heterogeneous and heteroscedastic mixtures , 2011 .

[19]  Varit Chaisinthop,et al.  Centralized and Distributed Semiparametric Compression of Piecewise Smooth Functions , 2011, IEEE Transactions on Signal Processing.

[20]  Aryeh Kontorovich,et al.  Model Selection for Sinusoids in Noise: Statistical Analysis and a New Penalty Term , 2011, IEEE Transactions on Signal Processing.

[21]  Yonina C. Eldar,et al.  Identification of Parametric Underspread Linear Systems and Super-Resolution Radar , 2010, IEEE Transactions on Signal Processing.

[22]  E. Candès,et al.  Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism , 2010, 1007.1434.

[23]  Yonina C. Eldar,et al.  Multichannel Sampling of Pulse Streams at the Rate of Innovation , 2010, IEEE Transactions on Signal Processing.

[24]  Yonina C. Eldar,et al.  Innovation Rate Sampling of Pulse Streams With Application to Ultrasound Imaging , 2010, IEEE Transactions on Signal Processing.

[25]  Yu. I. Ingster,et al.  Detection boundary in sparse regression , 2010, 1009.1706.

[26]  Yonina C. Eldar,et al.  Time-Delay Estimation From Low-Rate Samples: A Union of Subspaces Approach , 2009, IEEE Transactions on Signal Processing.

[27]  P. Hall,et al.  Innovated Higher Criticism for Detecting Sparse Signals in Correlated Noise , 2009, 0902.3837.

[28]  Pier Luigi Dragotti,et al.  Exact Feature Extraction Using Finite Rate of Innovation Principles With an Application to Image Super-Resolution , 2009, IEEE Transactions on Image Processing.

[29]  Patrick Rubin-Delanchy,et al.  Kinematics of Complex-Valued Time Series , 2008, IEEE Transactions on Signal Processing.

[30]  E. Candès,et al.  Searching for a trail of evidence in a maze , 2007, math/0701668.

[31]  O. Yli-Harja,et al.  Robust Fisher's Test for Periodicity Detection in Noisy Biological Time Series , 2007, 2007 IEEE International Workshop on Genomic Signal Processing and Statistics.

[32]  Jiashun Jin,et al.  Estimation and Confidence Sets for Sparse Normal Mixtures , 2006, math/0612623.

[33]  V. Koltchinskii,et al.  Concentration inequalities and asymptotic results for ratio type empirical processes , 2006, math/0606788.

[34]  Jie Chen,et al.  Bioinformatics Original Paper Detecting Periodic Patterns in Unevenly Spaced Gene Expression Time Series Using Lomb–scargle Periodograms , 2022 .

[35]  E. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[36]  Ronald K. Pearson,et al.  BMC Bioinformatics BioMed Central Methodology article , 2005 .

[37]  Xiaoming Huo,et al.  Near-optimal detection of geometric objects by fast multiscale methods , 2005, IEEE Transactions on Information Theory.

[38]  Korbinian Strimmer,et al.  Identifying periodically expressed transcripts in microarray time series data , 2008, Bioinform..

[39]  D. Donoho,et al.  Higher criticism for detecting sparse heterogeneous mixtures , 2004, math/0410072.

[40]  Dharmendra Lingaiah,et al.  The Estimation and Tracking of Frequency , 2004 .

[41]  Thierry Blu,et al.  Sampling signals with finite rate of innovation , 2002, IEEE Trans. Signal Process..

[42]  Qi-Man Shao,et al.  A normal comparison inequality and its applications , 2002 .

[43]  David L. Donoho,et al.  Application of basis pursuit in spectrum estimation , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[44]  W. Fuller,et al.  Introduction to Statistical Time Series (2nd ed.) , 1997 .

[45]  Petar M. Djuric,et al.  A model selection rule for sinusoids in white Gaussian noise , 1996, IEEE Trans. Signal Process..

[46]  Peter J. Kootsookos,et al.  Threshold behavior of the maximum likelihood estimator of frequency , 1994, IEEE Trans. Signal Process..

[47]  S. Kay Fundamentals of statistical signal processing: estimation theory , 1993 .

[48]  D. Donoho Superresolution via sparsity constraints , 1992 .

[49]  Richard A. Davis,et al.  Time Series: Theory and Methods (2nd ed.). , 1992 .

[50]  Tapan K. Sarkar,et al.  Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise , 1990, IEEE Trans. Acoust. Speech Signal Process..

[51]  Thomas Kailath,et al.  ESPRIT-estimation of signal parameters via rotational invariance techniques , 1989, IEEE Trans. Acoust. Speech Signal Process..

[52]  Shean-Tsong Chiu,et al.  Detecting Periodic Components in a White Gaussian Time Series , 1989 .

[53]  James A. Cadzow,et al.  Signal enhancement-a composite property mapping algorithm , 1988, IEEE Trans. Acoust. Speech Signal Process..

[54]  K. Alexander,et al.  Rates of growth and sample moduli for weighted empirical processes indexed by sets , 1987 .

[55]  R. Davies Hypothesis Testing when a Nuisance Parameter is Present Only Under the Alternatives , 1987 .

[56]  Alfred M. Bruckstein,et al.  The resolution of overlapping echos , 1985, IEEE Trans. Acoust. Speech Signal Process..

[57]  M. R. Leadbetter,et al.  Extremes and Related Properties of Random Sequences and Processes: Springer Series in Statistics , 1983 .

[58]  J. Scargle Studies in astronomical time series analysis. II - Statistical aspects of spectral analysis of unevenly spaced data , 1982 .

[59]  S.M. Kay,et al.  Digital signal processing for sonar , 1981, Proceedings of the IEEE.

[60]  J. Scargle Studies in astronomical time series analysis. I - Modeling random processes in the time domain , 1981 .

[61]  Andrew F. Siegel,et al.  Testing for Periodicity in a Time Series , 1980 .

[62]  R. O. Schmidt,et al.  Multiple emitter location and signal Parameter estimation , 1986 .

[63]  M. Skolnik,et al.  Introduction to Radar Systems , 2021, Advances in Adaptive Radar Detection and Range Estimation.

[64]  Martien C. A. van Zuijlen,et al.  Properties of the Empirical Distribution Function for Independent Non- Identically Distributed Random Vectors , 1978 .

[65]  R. Davies Hypothesis testing when a nuisance parameter is present only under the alternative , 1977 .

[66]  Ван Трис Гарри Теория обнаружения, оценок и модуляции. Обработка сигналов в радио- и гидролокации и прием случайных гауссовых сигналов на фоне помех. (Detection, Estimation, and Modulation Theory. P.III. Radar-Sonar Signal Processing and Gaussian Signals in Noise) , 1977 .

[67]  Chris Chatfield,et al.  Introduction to Statistical Time Series. , 1976 .

[68]  N. Lomb Least-squares frequency analysis of unequally spaced data , 1976 .

[69]  E. Bølviken TESTS OF SIGNIFICANCE IN PERIODOGRAM ANALYSIS , 1976 .

[70]  Harry L. Van Trees,et al.  Detection, Estimation, and Modulation Theory: Radar-Sonar Signal Processing and Gaussian Signals in Noise , 1992 .

[71]  Robert N. McDonough,et al.  Detection of signals in noise , 1971 .

[72]  James Durbin,et al.  Tests for serial correlation in regression analysis based on the periodogram of least-squares residuals , 1969 .

[73]  H. Hartley,et al.  Tests of significance in harmonic analysis. , 1949, Biometrika.