A comparative study of the reliability of nine statistical software packages

The reliabilities of nine software packages commonly used in performing statistical analysis are assessed and compared. The (American) National Institute of Standards and Technology (NIST) data sets are used to evaluate the performance of these software packages with regard to univariate summary statistics, one-way ANOVA, linear regression, and nonlinear regression. Previous research has examined various versions of these software packages using the NIST data sets, but typically with fewer software packages than used in this study. This study provides insight into a relative comparison of a wide variety of software packages including two free statistical software packages, basic and advanced statistical software packages, and the popular Excel package. Substantive improvements from previous software reliability assessments are noted. Plots of principal components of a measure of the correct number of significant digits reveal how these packages tend to cluster for ANOVA and nonlinear regression.

[1]  William S. Cleveland,et al.  Visualizing Data , 1993 .

[2]  B. D. McCULLOUGH.  A REVIEW OF TESTU 01 , 2006 .

[3]  B. D. McCullough,et al.  On the accuracy of statistical procedures in Microsoft Excel 2003 , 2005, Comput. Stat. Data Anal..

[4]  P. Fayers,et al.  The Visual Display of Quantitative Information , 1990 .

[5]  Leo Kn usel.  On the accuracy of statistical distributions in Microsoft Excel 2003 , 1998 .

[6]  Micah Altman,et al.  Numerical Issues in Statistical Computing for the Social Scientist , 2003 .

[7]  Bruce D. McCullough,et al.  On the accuracy of statistical procedures in Microsoft Excel 2000 and Excel XP , 2002 .

[8]  Günther Sawitzki,et al.  Report on the numerical reliability of data analysis systems , 1994 .

[9]  Bruce D. McCullough,et al.  A review of TESTU01 , 2006 .

[10]  B. D. McCullough,et al.  The accurary of Mathematica 4 as a statistical package , 2000, Comput. Stat..

[11]  Leo Knüsel,et al.  On the accuracy of statistical distributions in Microsoft Excel 97 , 1998 .

[12]  Bruce D. McCullough,et al.  Is it safe to assume that software is accurate , 2000 .

[13]  B. D. McCullough.  The accurary of , 2000 .

[14]  Jürgen Symanzik,et al.  Assessing the reliability of web-based statistical software , 2003, Comput. Stat..

[15]  Micah Altman,et al.  Numerical Issues in Statistical Computing for the Social Scientist , 2003 .

[16]  Bruce D. McCullough,et al.  Assessing the Reliability of Statistical Software: Part I , 1998 .

[17]  D R Griffin,et al.  Letters to the editor. , 1974, Science.

[18]  Matthew W. Rohrer.  Seeing is believing: the importance of visualization in manufacturing simulation , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[19]  Roger L. Berger,et al.  Lehr, R. G. (2000), Letter to the editor, the American statistician, 54, 325: Comment by berger and doi and reply , 2001 .

[20]  Bruce D. McCullough,et al.  Wilkinson's Tests and Econometric Software , 1999 .

[21]  Micah Altman,et al.  Choosing Reliable Statistical Software , 2001, PS: Political Science & Politics.

[22]  David A. Heiser,et al.  On the accuracy of statistical procedures in Microsoft Excel 2007 , 1999, Comput. Stat. Data Anal..

[23]  Leon S. Lasdon,et al.  Design and Use of the Microsoft Excel Solver , 1998, Interfaces.

[24]  Edward R. Tufte,et al.  The Visual Display of Quantitative Information , 1986 .

[25]  Bruce D. McCullough,et al.  Econometric Software Reliability: EViews, LIMDEP, SHAZAM and TSP , 1999 .

[26]  Jonathan D. Cryer,et al.  Problems With Using Microsoft Excel for Statistics , 2001 .