Adaptive Threshold Estimation via Extreme Value Theory

Determining a detection threshold to automatically maintain a low false alarm rate is a challenging problem. In a number of different applications, the underlying parametric assumptions of most automatic target detection algorithms are invalid. Therefore, thresholds derived using these incorrect distribution assumptions do not produce desirable results when applied to real sensor data. Monte Carlo methods for threshold determination work well but tend to perform poorly when targets are present. In order to mitigate these effects, we propose an algorithm using extreme value theory through the use of the generalized Pareto distribution (GPD) and a Kolmogorov-Smirnov statistical test. Unlike previous work based on GPD estimates, this algorithm incorporates a way to adaptively maintain low false alarm rates in the presence of targets. Both synthetic and real-world detection results demonstrate the usefulness of this algorithm.

[1]  D. Siegmund.  Importance Sampling in the Monte Carlo Study of Sequential Tests , 1976 .

[2]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[3]  Dimitris Manolakis,et al.  Realistic matched filter performance prediction for hyperspectral target detection , 2004, IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium.

[4]  Maurizio Longo,et al.  Comparative performance analysis of some extrapolative estimators of probability tails , 1988, IEEE J. Sel. Areas Commun..

[5]  Jeffrey C. Lagarias,et al.  Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions , 1998, SIAM J. Optim..

[6]  Louis L. Scharf,et al.  The adaptive coherence estimator: a uniformly most-powerful-invariant adaptive detection statistic , 2005, IEEE Transactions on Signal Processing.

[7]  Joshua B. Broadwater,et al.  Physics-Based Detection of Subpixel Targets in Hyperspectral Imagery , 2007 .

[8]  Rama Chellappa,et al.  Hybrid Detectors for Subpixel Targets , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Ramazan Gençay,et al.  EVIM: A Software Package for Extreme Value Analysis in MATLAB , 2001 .

[10]  H. Lilliefors.  On the Kolmogorov-Smirnov Test for Normality with Mean and Variance Unknown , 1967 .

[11]  James A. Bucklew.  The Blind Simulation Problem and Regenerative Processes , 1998, IEEE Trans. Inf. Theory.

[12]  Louis L. Scharf,et al.  Adaptive subspace detectors , 2001, IEEE Trans. Signal Process..

[13]  L. Haan,et al.  Residual Life Time at Great Age , 1974 .

[14]  J.S. Sadowsky,et al.  On large deviations theory and asymptotically efficient Monte Carlo estimation , 1990, IEEE Trans. Inf. Theory.

[15]  Laurence B. Milstein,et al.  Robust detection using extreme-value theory , 1969, IEEE Trans. Inf. Theory.

[16]  J. Pickands.  Statistical Inference Using Extreme Order Statistics , 1975 .

[17]  M. Kendall,et al.  Kendall's advanced theory of statistics , 1995 .

[18]  W. DuMouchel.  Estimating the Stable Index $\alpha$ in Order to Measure Tail Thickness: A Critique , 1983 .

[19]  S. Roberts.  Novelty detection using extreme value statistics , 1999 .

[20]  Robert P. W. Duin,et al.  Support vector domain description , 1999, Pattern Recognit. Lett..

[21]  E. M. Winter,et al.  Anomaly detection from hyperspectral imagery , 2002, IEEE Signal Process. Mag..

[22]  B. Gnedenko.  Sur La Distribution Limite Du Terme Maximum D'Une Serie Aleatoire , 1943 .

[23]  Richard L. Smith.  Extreme Value Analysis of Environmental Time Series: An Application to Trend Detection in Ground-Level Ozone , 1989 .

[24]  Anthony C. Davison,et al.  Statistics of Extremes , 2015, International Encyclopedia of Statistical Science.

[25]  Dimitris G. Manolakis,et al.  Detection algorithms for hyperspectral imaging applications , 2002, IEEE Signal Process. Mag..

[26]  R. Srinivasan.  Importance Sampling: Applications in Communications and Detection , 2010 .

[27]  PAUL EMBRECHTS,et al.  Modelling of extremal events in insurance and finance , 1994, Math. Methods Oper. Res..

[28]  R. Fisher,et al.  Limiting forms of the frequency distribution of the largest or smallest member of a sample , 1928, Mathematical Proceedings of the Cambridge Philosophical Society.

[29]  P.R. Chakravarthi,et al.  On determining the radar threshold for non-Gaussian processes from experimental data , 1996, IEEE Trans. Inf. Theory.

[30]  R. Srinivasan,et al.  Simulation of CFAR detection algorithms for arbitrary clutter distributions , 2000 .

[31]  Stephen B. Weinstein,et al.  Theory and application of some classical and generalized asymptotic distributions of extreme values , 1973, IEEE Trans. Inf. Theory.

[32]  R. Kay.  The Analysis of Survival Data , 2012 .

[33]  J. Corcoran.  Modelling Extremal Events for Insurance and Finance , 2002 .

[34]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[35]  C. Klüppelberg,et al.  Modelling Extremal Events , 1997 .

[36]  J.N. Briggs.  Target detection by marine radar , 2005, IEEE Aerospace and Electronic Systems Magazine.

[37]  Dimitris G. Manolakis,et al.  Maintaining CFAR operation in hyperspectral target detection using extreme value distributions , 2007, SPIE Defense + Commercial Sensing.