Determining output uncertainty of computer system models

Computer system models provide detailed answers to system performance. For a given system configuration, a system model estimates the cycles per instruction that the system would incur while running a given workload. In addition, it estimates the proportion of time that is spent in different parts of the system and other related metrics such as bus utilizations. Consider those inputs to a system model that are estimated with uncertainty. Examples include cache miss rates that are obtained via trace-driven cache simulation and also sometimes by extrapolating beyond the simulation domain. Errors incurred during the measurement and fitting processes are propagated to the system model outputs. On the other hand, other inputs such as hardware latencies are known precisely. In this paper we propose several measures of uncertainty of system model outputs when it stems from uncertainty in the inputs. Some of these measures are based on sensitivity of an output to the inputs. We propose ways of defining and determining these sensitivities and turning them into uncertainty measures. Other measures are based on sampling schemes. Additionally, we determine uncertainty measures for the system model outputs over a wide range of inputs covering large system design spaces. This is done by first selecting a set of input configurations based on an experimental design methodology where the uncertainty measures are determined. Then these data are used to interpolate the uncertainty measure function over the rest of the input space. We quantitatively characterize each input's contribution to the output uncertainty over the input's entire range. We also propose ways that call attention to high output uncertainty regions in the input space. The methodology is illustrated on system models developed at Sun Microsystems Laboratories. The particular goal of the performance analysis is a design of level two caches.

[1]  Pradip Bose,et al.  Performance Analysis and Its Impact on Design , 1998, Computer.

[2]  Ralph Haas,et al.  Sensitivity Analysis of Computer Models: World Bank HDM-III Model , 1999 .

[3]  Günter Haring,et al.  Performance Bounds for Distributed Systems with Workload Variabilities and Uncertainties , 1997, Parallel Comput..

[4]  Jack P. C. Kleijnen,et al.  Sensitivity analysis versus uncertainty analysis: when to use what? in predictability and nonlinear , 1994 .

[5]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[6]  Jianqing Fan,et al.  Local polynomial modelling and its applications , 1994 .

[7]  Ronald W. Wolff,et al.  Stochastic Modeling and the Theory of Queues , 1989 .

[8]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[9]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[10]  Mark Horowitz,et al.  An analytical cache model , 1989, TOCS.

[11]  Wen-Hann Wang,et al.  On the inclusion properties for multi-level cache hierarchies , 1988, ISCA '88.

[12]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[13]  Douglas M. Hawkins,et al.  A statistically rigorous approach for improving simulation methodology , 2003, The Ninth International Symposium on High-Performance Computer Architecture, 2003. HPCA-9 2003. Proceedings..

[14]  Art B. Owen,et al.  9 Computer experiments , 1996, Design and analysis of experiments.

[15]  Ilya Gluhovsky,et al.  Comprehensive multiprocessor cache miss rate generation using multivariate models , 2005, TOCS.

[16]  Szu Hui Ng,et al.  Reducing input parameter uncertainty for simulations , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[17]  David A. Wood,et al.  Variability in architectural simulations of multi-threaded workloads , 2003, The Ninth International Symposium on High-Performance Computer Architecture, 2003. HPCA-9 2003. Proceedings..

[18]  James R. Wilson,et al.  Accounting for Parameter Uncertainty in Simulation Input Modeling , 2003 .

[19]  Mikko H. Lipasti,et al.  A performance methodology for commercial servers , 2000, IBM J. Res. Dev..

[20]  T. J. Mitchell,et al.  Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments , 1991 .

[21]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[22]  R. Cheng,et al.  Two-point methods for assessing variability in simulation output , 1998 .