Examining the reliability of logistic regression estimation software

The reliability of nine software packages using the maximum likelihood estimator for the logistic regression model were examined using generated benchmark datasets and models. Software packages tested included: SAS (Procs Logistic, Catmod, Genmod, Surveylogistic, Glimmix, and Qlim), Limdep (Logit, Blogit), Stata (Logit, GLM, Binreg), Matlab, Shazam, R, Minitab, Eviews, and SPSS for all available algorithms, none of which have been previously tested. This study expands on the existing literature in this area by examination of Minitab 15 and SPSS 17. The findings indicate that Matlab, R, Eviews, Minitab, Limdep (BFGS), and SPSS provided consistently reliable results for both parameter and standard error estimates across the benchmark datasets. While some packages performed admirably, shortcomings did exist. SAS maximum log-likelihood estimators do not always converge to the optimal solution and stop prematurely depending on starting values, by issuing a "flat" error message. This drawback can be dealt with by rerunning the maximum log-likelihood estimator, using a closer starting point, to see if the convergence criteria are actually satisfied. Although Stata-Binreg provides reliable parameter estimates, there is no way to obtain standard error estimates in Stata-Binreg as of yet. Limdep performs relatively well, but did not converge due to a weakness of the algorithm. The results show that solely trusting the default settings of statistical software packages may lead to non-optimal, biased or erroneous results, which may impact the quality of empirical results obtained by applied economists. Reliability tests indicate severe weaknesses in SAS Procs Glimmix and Genmod. Some software packages fail reliability tests under certain conditions. The finding indicates the need to use multiple software packages to solve econometric models.

[1]  Arturo Estrella,et al.  A new measure of fit for equations with dichotomous dependent variables , 1998 .

[2]  P. Schmidt,et al.  Limited-Dependent and Qualitative Variables in Econometrics. , 1984 .

[3]  Sunil J Rao,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2003 .

[4]  Micah Altman,et al.  Choosing Reliable Statistical Software , 2001, PS: Political Science & Politics.

[5]  James Demmel,et al.  Design, implementation and testing of extended and mixed precision BLAS , 2000, TOMS.

[6]  Luis A. Escobar,et al.  Teaching about Approximate Confidence Regions Based on Maximum Likelihood Estimation , 1995 .

[7]  Arnold Zellner,et al.  JOINT ESTIMATION OF RELATIONSHIPS INVOLVING DISCRETE RANDOM VARIABLES , 1965 .

[8]  A. Talha Yalta,et al.  On the importance of verifying forecasting results , 2009 .

[9]  William N. Venables,et al.  Modern Applied Statistics with S-Plus. , 1996 .

[10]  H. Vinod.  Review of GAUSS for Windows, including its numerical accuracy , 2000 .

[11]  J. C. Gower,et al.  Accuracy and stability , 2004 .

[12]  Xiaoye S. Li,et al.  Algorithms for quad-double precision floating point arithmetic , 2000, Proceedings 15th IEEE Symposium on Computer Arithmetic. ARITH-15 2001.

[13]  Leo Knüsel,et al.  On the accuracy of statistical distributions in Microsoft Excel 97 , 1998 .

[14]  Bruce D. McCullough,et al.  On the accuracy of statistical procedures in Microsoft Excel 2000 and Excel XP , 2002 .

[15]  W. Hauck,et al.  Wald's Test as Applied to Hypotheses in Logit Analysis , 1977 .

[16]  Robert A. Oster,et al.  An Examination of Statistical Software Packages for Categorical Data Analysis Using Exact Methods , 2002 .

[17]  F. J. Anscombe,et al.  Topics in the Investigation of Linear Relations Fitted by the Method of Least Squares , 1967 .

[18]  Chuhsing Kate Hsiao,et al.  Nonlinear regression analysis , 2010 .

[19]  Roger Fletcher,et al.  Practical methods of optimization; (2nd ed.) , 1987 .

[20]  Eberechukwu Onukwugha,et al.  Bernoulli Regression Models: Revisiting the Specification of Statistical Models with Binary Dependent Variables , 2010 .

[21]  Micah Altman,et al.  accuracy: Tools for Accurate and Reliable Statistical Computing , 2007 .

[22]  James W. Longley.  An Appraisal of Least Squares Programs for the Electronic Computer from the Point of View of the User , 1967 .

[23]  Bruce D. McCullough,et al.  The Numerical Reliability of Econometric Software , 1999 .

[24]  Günther Sawitzki,et al.  Report on the numerical reliability of data analysis systems , 1994 .

[25]  J. Hilbe,et al.  An Examination of Statistical Software Packages for Parametric and Nonparametric Data Analyses Using Exact Methods , 2008 .

[26]  Charles G. Renfro.  A compendium of existing econometric software packages , 2004 .

[27]  S. Cessie,et al.  Ridge Estimators in Logistic Regression , 1992 .

[28]  Michael L. Overton,et al.  Numerical Computing with IEEE Floating Point Arithmetic , 2001 .

[29]  Robert B. Schnabel,et al.  Computational experience with confidence intervals for nonlinear least squares , 1986 .

[30]  S Greenland,et al.  Problems due to small samples and sparse data in conditional logistic regression analysis. , 2000, American journal of epidemiology.

[31]  David A. Heiser,et al.  On the accuracy of statistical procedures in Microsoft Excel 2007 , 1999, Comput. Stat. Data Anal..

[32]  James P. LeSage,et al.  Numerical accuracy of statistical algorithms for microcomputers , 1985 .

[33]  Michael R. Veall,et al.  Shazam 6.2: A review , 1991 .

[34]  A. Albert,et al.  On the existence of maximum likelihood estimates in logistic regression models , 1984 .

[35]  Aris Spanos,et al.  The problem of near-multicollinearity revisited: erratic vs systematic volatility , 2002 .

[36]  Allen M. Featherstone,et al.  Reliability of Statistical Software , 2010 .

[37]  Micah Altman,et al.  Some Details of Nonlinear Estimation , 2004 .

[38]  Leon S. Lasdon,et al.  OR Practice - The Status of Nonlinear Programming Software: An Update , 1987, Oper. Res..

[39]  Micah Altman,et al.  Numerical Issues in Statistical Computing for the Social Scientist , 2003 .

[40]  Günther Sawitzki,et al.  Testing numerical reliability of data analysis systems , 1994 .

[41]  Charles G. Renfro,et al.  Some numerical aspects of nonlinear estimation , 2000 .

[42]  David A. Belsley,et al.  Regression Analysis and its Application: A Data-Oriented Approach.@@@Applied Linear Regression.@@@Regression Diagnostics: Identifying Influential Data and Sources of Collinearity , 1981 .

[43]  Paul D. Allison,et al.  Convergence Failures in Logistic Regression , 2008 .

[44]  Roy H. Wampler.  A Report on the Accuracy of Some Widely Used Least Squares Computer Programs , 1970 .

[45]  Charles G. Renfro,et al.  Benchmarks and software standards: A case study of GARCH procedures , 1998 .

[46]  Micah Altman,et al.  Numerical Issues in Statistical Computing for the Social Scientist , 2003 .

[47]  Robert J. Pavur,et al.  A comparative study of the reliability of nine statistical software packages , 2007, Comput. Stat. Data Anal..

[48]  Houston H. Stokes,et al.  On the advantage of using two or more econometric software systems to solve the same problem , 2004 .

[49]  Giorgio Calzolari,et al.  Alternative Estimators of FIML Covariance Matrix: A Monte Carlo Stud y , 1988 .

[50]  Leo Kn usel.  On the accuracy of statistical distributions in Microsoft Excel 2003 , 1998 .

[51]  Bruce D. McCullough,et al.  Assessing the Reliability of Statistical Software: Part I , 1998 .

[52]  James P. LeSage,et al.  Benchmarking numerical accuracy of statistical algorithms , 1988 .

[53]  I. W. Burr.  Cumulative Frequency Functions , 1942 .

[54]  Roy H. Wampler.  Test procedures and test problems for least squares algorithms , 1980 .

[55]  C. D. Beaumont,et al.  Regression Diagnostics — Identifying Influential Data and Sources of Collinearity , 1981 .

[56]  David B. Allison,et al.  How accurate are the extremely small P-values used in genomic research: An evaluation of numerical libraries , 2009, Comput. Stat. Data Anal..

[57]  H. Vinod,et al.  Verifying the Solution from a Nonlinear Solver: A Case Study , 2004 .

[58]  J. H. Wilkinson,et al.  AN ESTIMATE FOR THE CONDITION NUMBER OF A MATRIX , 1979 .

[59]  S. Addelman.  Orthogonal Main-Effect Plans for Asymmetrical Factorial Experiments , 1962 .

[60]  Douglas M. Bates,et al.  Nonlinear Regression Analysis and Its Applications , 1988 .

[61]  J. MacKinnon,et al.  Econometric Theory and Methods , 2003 .

[62]  Bruce D. McCullough,et al.  Econometric Software Reliability: EViews, LIMDEP, SHAZAM and TSP , 1999 .

[63]  A. T. Yalta.  The Numerical Reliability of GAUSS 8.0 , 2007 .

[64]  P. McCullagh,et al.  Generalized Linear Models, 2nd Edn. , 1990 .