Benchmarks and software standards: A case study of GARCH procedures

This paper addresses the evaluation of nonlinear methods in econometric software, taking GARCH procedures as a case study. In particular, it analyzes seven widely used packages, utilizing a recently developed benchmark. Four of the packages are found to be unsuitable, in most cases because the developer either does not specifically indicate which of the many possible GARCH models is being estimated, or does not accommodate the most common model specified in the applied literature, or both. A principal finding is that implementation of the GARCH procedure varies so widely that two packages ostensibly doing the same thing actually may be estimating substantively different models. This lack of standardization raises several questions concerning the evaluation of software. These include the issues normally associated with the creation of benchmarks, but also the critical role that software plays, and can play, in the development of modern econometrics.

[1]  Eric Ghysels,et al.  Periodic Autoregressive Conditional Heteroskedasticity , 1996 .

[2]  M GayDavid,et al.  Algorithm 611: Subroutines for Unconstrained Minimization Using a Model/Trust-Region Approach , 1983 .

[3]  Gabriele Fiorentini,et al.  Analytic Derivatives and the Computation of Garch Estimates , 1996 .

[4]  T. Brubaker,et al.  Nonlinear Parameter Estimation , 1979 .

[5]  T. Bollerslev,et al.  Generalized autoregressive conditional heteroskedasticity , 1986 .

[6]  Bruce D. McCullough,et al.  A Review of RATS v4.2: Benchmarking Numerical Accuracy , 1997 .

[7]  Giorgio Calzolari,et al.  Alternative Estimators of FIML Covariance Matrix: A Monte Carlo Stud y , 1988 .

[8]  David M. Gay,et al.  Algorithm 611: Subroutines for Unconstrained Minimization Using a Model/Trust-Region Approach , 1983, TOMS.

[9]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[10]  Charles G. Renfro.  Normative considerations in the development of a software package for econometric estimation , 1997 .

[11]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[12]  Christina Fraley.  Software Performance on Nonlinear Least-Squares Problems , 1989 .

[13]  Michael C. Lovell,et al.  Econometric Software Accidents , 1994 .

[14]  Robert B. Schnabel,et al.  Computational experience with confidence intervals for nonlinear least squares , 1986 .

[15]  Julian Silk,et al.  Systems Estimation: A Comparison of SAS, SHAZAM and TSP , 1996 .

[16]  H. Iemoto.  Modelling the persistence of conditional variances , 1986 .

[17]  Jerry G. Thursby,et al.  Replication in Empirical Economics: The Journal of Money, Credit and Banking Project , 1986 .

[18]  James W. Longley.  An Appraisal of Least Squares Programs for the Electronic Computer from the Point of View of the User , 1967 .

[19]  Marcel G. Dagenais,et al.  The Computation of FIML Estimates as Iterative Generalized Least Squares Estimates in Linear and Nonlinear Simultaneous Equations Models , 1978 .

[20]  R. Lumsdaine,et al.  Consistency and Asymptotic Normality of the Quasi-maximum Likelihood Estimator in IGARCH(1,1) and Covariance Stationary GARCH(1,1) Models , 1996 .

[21]  Philip E. Gill,et al.  Practical optimization , 1981 .

[22]  Tim Bollerslev,et al.  Chapter 49 Arch models , 1994 .

[23]  Christoph W. Ueberhuber,et al.  Numerical Computation 2 , 1997 .

[24]  Richard E. Quandt,et al.  COMPUTATIONAL PROBLEMS AND METHODS , 1983 .

[25]  Til Schuermann,et al.  Exact Maximum Likelihood Estimation of Observation-Driven Econometric Models , 1996 .

[26]  Susan Feigenbaum,et al.  The market for (Ir)reproducible econometrics , 1993 .