A Technique with Low Memory and Computational Requirements for Dynamic Tracking of Quantiles

A technique for dynamic tracking of quantiles of data streams, without storage and sorting of past data samples, is presented. It updates the quantile estimate recursively by applying an increment, selected as a fraction of the range, such that the estimated quantile approaches the sample quantile. The range is dynamically estimated using first-order recursive relations for peak and valley detection. The technique does not require initial estimates and the computation steps involved are the same for all the samples. It has low memory and computational requirements and is suitable for signal processing and other applications involving online tracking of single or multiple quantiles of data streams. It has been tested using synthetic and real data with different distributions.

[1]  Baba Thiam,et al.  A smoothing stochastic algorithm for quantile estimation , 2014 .

[2]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[3]  Sanjay Ranka,et al.  A One-Pass Algorithm for Accurately Estimating Quantiles for Disk-Resident Data , 1997, VLDB.

[4]  Bruce G. Lindsay,et al.  Approximate medians and other quantiles in one pass and with limited memory , 1998, SIGMOD '98.

[5]  David Pearce,et al.  The aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions , 2000, INTERSPEECH.

[6]  Nitya Tiwari,et al.  Speech enhancement using noise estimation based on dynamic quantile tracking for hearing impaired listeners , 2015, 2015 Twenty First National Conference on Communications (NCC).

[7]  Fei Chen,et al.  Incremental quantile estimation for massive tracking , 2000, KDD '00.

[8]  Divesh Srivastava,et al.  Effective computation of biased quantiles over data streams , 2005, 21st International Conference on Data Engineering (ICDE'05).

[9]  Alexander Fischer,et al.  Quantile based noise estimation for spectral subtraction and Wiener filtering , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[10]  Bruce G. Lindsay,et al.  Random sampling techniques for space efficient online computation of order statistics of large datasets , 1999, SIGMOD '99.

[11]  Lu Wang,et al.  Quantiles over data streams: an experimental study , 2013, SIGMOD '13.

[12]  Sudipto Guha,et al.  Data-streams and histograms , 2001, STOC '01.

[13]  Sudipto Guha,et al.  Lower Bounds for Quantile Estimation in Random-Order and Multi-pass Streaming , 2007, ICALP.

[14]  KhannaSanjeev,et al.  Space-efficient online computation of quantile summaries , 2001 .

[15]  John Mason,et al.  Time-Frequency Quantile-Based Noise Estimation , 2002 .

[16]  W. David Kelton,et al.  Density estimation from correlated data , 2014, J. Simulation.

[17]  Gurmeet Singh Manku,et al.  Approximate counts and quantiles over sliding windows , 2004, PODS.

[18]  Christophe Ris,et al.  Assessing local noise level estimation methods: Application to noise robust ASR , 2000, Speech Commun..

[19]  J. Ian Munro,et al.  Selection and sorting with limited storage , 1978, 19th Annual Symposium on Foundations of Computer Science (sfcs 1978).

[20]  Diane Lambert,et al.  Monitoring Networked Applications With Incremental Quantile Estimation , 2006, 0708.0302.

[21]  Imrich Chlamtac,et al.  The P2 algorithm for dynamic calculation of quantiles and histograms without storing observations , 1985, CACM.

[22]  Graham Cormode,et al.  An improved data stream summary: the count-min sketch and its applications , 2004, J. Algorithms.

[23]  Sudipto Guha,et al.  Stream Order and Order Statistics: Quantile Estimation in Random-Order Streams , 2009, SIAM J. Comput..

[24]  Luke Tierney,et al.  A Space-Efficient Recursive Procedure for Estimating a Quantile of an Unknown Distribution , 1983 .

[25]  Incremental tracking of multiple quantiles for network monitoring in cellular networks , 2009, MICNET '09.

[26]  Rakesh Agrawal,et al.  A One-Pass Space-Efficient Algorithm for Finding Quantiles , 1995, COMAD.

[27]  Eva Möller,et al.  Statistical Properties and Control Algorithms of Recursive Quantile Estimators , 2000 .

[28]  Sanjeev Khanna,et al.  Space-efficient online computation of quantile summaries , 2001, SIGMOD '01.

[29]  W. David Kelton,et al.  Estimating steady-state distributions via simulation-generated histograms , 2008, Comput. Oper. Res..