Extraction of unknown signals in arbitrary noise.

We devise a general method to extract weak signals of unknown form, buried in noise of arbitrary distribution. Central to it is signal-noise decomposition in rank and time: only stationary white noise generates data with a jointly uniform rank-time probability distribution, U(1,N)×U(1,N), for N points in a data sequence. We show that rank, averaged across jointly indexed series of noisy data, tracks the underlying weak signal via a simple relation, for all noise distributions. We derive an exact analytic, distribution-independent form for the discrete covariance matrix of cumulative distributions for independent and identically distributed noise and employ its eigenfunctions to extract unknown signals from single time series.

[1]  P. Embrechts,et al.  Linear Regression for Heavy Tails , 2018, Risks.

[2]  T. Srokowski Multiplicative Lévy noise in bistable systems , 2012, 1202.3023.

[3]  Noise: From nuisance to research subject , 2020 .

[4]  Martin Wild,et al.  Global dimming and brightening: A review , 2009 .

[5]  Stephen M. Stigler,et al.  The Seven Pillars of Statistical Wisdom , 2016 .

[6]  Miguel Herrero-Collantes,et al.  Quantum random number generators , 2016, 1604.03304.

[7]  A. C. Koivunen,et al.  The Feasibility of Data Whitening to Improve Performance of Weather Radar , 1999 .

[8]  J. Klafter,et al.  First Steps in Random Walks: From Tools to Applications , 2011 .

[9]  S. Redner,et al.  A Kinetic View of Statistical Physics , 2010 .

[10]  Glenn Ierley,et al.  Universal Rank-Order Transform to Extract Signals from Noisy Data , 2019, Physical Review X.

[11]  Christian Beck,et al.  Superstatistical approach to air pollution statistics , 2020, Physical Review Research.

[12]  Jean-Philippe Bouchaud,et al.  Weighted Kolmogorov-Smirnov test: accounting for the tails. , 2012, Physical review. E, Statistical, nonlinear, and soft matter physics.

[13]  R. Dong,et al.  A generator for unique quantum random numbers based on vacuum states , 2010 .

[14]  Congzhang Gao,et al.  Stochastic radiative transfer in random media: Pure absorbing cases. , 2020, Physical review. E.

[15]  Liu Xingzhao,et al.  Cauchy pdf modelling and its application to SAR image despeckling , 2008 .

[16]  Robert C. Dalang,et al.  L\'evy Processes and L\'evy White Noise as Tempered Distributions , 2015, 1509.05274.

[17]  S. Radošević,et al.  Devil's staircase and the absence of chaos in the dc- and ac-driven overdamped Frenkel-Kontorova model. , 2017, Physical review. E.

[18]  S. Majumdar,et al.  Record-breaking statistics for random walks in the presence of measurement error and noise. , 2013, Physical review letters.

[19]  Pramod K. Varshney,et al.  Linear Detection of a Weak Signal in Additive Cauchy Noise , 2017, IEEE Transactions on Communications.

[20]  Masahito Ueda,et al.  Standard Quantum Limit and Heisenberg Limit in Function Estimation. , 2018, Physical review letters.

[22]  R. Filip,et al.  Indefinite-Mean Pareto Photon Distribution from Amplified Quantum Noise. , 2018, Physical review letters.