System Understanding and Statistical Uncertainty Bounds from Limited Test Data

n many DoD test and evaluation programs, it is necessary to obtain statistical estimates for parameters in the system under study. For these estimates to provide meaningful system understanding, uncertainty bounds (e.g., statistical confidence intervals) must be attached to the estimates. Current methods for constructing uncertainty bounds are almost all based on theory that assumes a large amount of test data. Such methods are not justified in many realistic testing environments where only a limited amount of data is available. This article presents a new method for constructing uncertainty bounds for a broad class of statistical estimation procedures when faced with only a limited amount of data. The approach is illustrated on a problem motivated by a Navy program related to missile accuracy, where each test is very expensive. This example will illustrate how the small-sample approach is able to obtain more information from the limited sample than traditional approaches such as asymptotic approximations and the bootstrap.

[1]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[2]  Jayanta K. Ghosh,et al.  Higher Order Asymptotics , 1994 .

[3]  T. Anderson.  Statistical analysis of time series , 1974 .

[4]  Saddlepoint Approximations in the Case of Intractable Cumulant Generating Functions , 1991 .

[5]  D. A. Shnidman,et al.  Efficient computation of the circular error probability (CEP) integral , 1995, IEEE Trans. Autom. Control..

[6]  Ronald W. Butler,et al.  Saddlepoint Approximations to the CDF of Some Statistics with Nonnormal Limit Distributions , 1993 .

[7]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[8]  Nancy Reid,et al.  Saddlepoint Methods and Statistical Inference , 1988 .

[9]  J. L. Maryak,et al.  A Feasible Bayesian Estimator of Quantiles for Projectile Accuracy From Non-iid Data , 1992 .

[10]  J. Spall.  Uncertainty bounds for parameter identification with small sample sizes , 1995, Proceedings of 1995 34th IEEE Conference on Decision and Control.

[11]  C. Gouriéroux.  Estimation and Tests , 1997 .

[12]  Jan Kmenta,et al.  Elements of econometrics , 1988 .

[13]  Richard H. Smith.  Maximum likelihood mean and covariance matrix estimation constrained to general positive semi-definiteness , 1985 .

[14]  Elvezio Ronchetti,et al.  Empirical Saddlepoint Approximations for Multivariate M-estimators , 1994 .

[15]  J. Berger,et al.  Empirical Bayes Estimation of Rates in Longitudinal Studies , 1983 .

[16]  W. F. Trench,et al.  Multivariable Calculus with Linear Algebra and Series , 1972 .

[17]  S. Rice,et al.  Saddle point approximation for the distribution of the sum of independent random variables , 1980, Advances in Applied Probability.

[18]  Silvelyn Zwanzig,et al.  Discussion to A. Pazman: Small-sample distributional properties of nonlinear regression estimator (a geometric approach) , 1990 .

[19]  Chris Field,et al.  Small sample confidence intervals , 1990 .

[20]  H. Daniels.  Saddlepoint Approximations in Statistics , 1954 .

[21]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[22]  L. Ljung.  Convergence analysis of parametric identification methods , 1978 .

[23]  R. Serfling.  Approximation Theorems of Mathematical Statistics , 1980 .

[24]  Matrix Weighting of Several Regression Coefficient Vectors , 1993 .

[25]  R. Shumway,et al.  Estimation and tests of hypotheses for the initial mean and covariance in the kalman filter model , 1981 .

[26]  Fang-Kuo Sun,et al.  A maximum likelihood algorithm for the mean and covariance of nonidentically distributed observations , 1982 .

[27]  Jack Kaplan,et al.  Estimators for the One-Way Random Effects Model with Unequal Error Variances , 1981 .

[28]  James C. Spall,et al.  First-order data sensitivity measures with applications to a multivariate signal-plus-noise problem , 1990 .