Report on the numerical reliability of data analysis systems

Abstract From 1990 to 1993, a series of test on numerical reliability of data analysis systems has been carried out. The tests are based on L. Wilkinson's “Statistics Quiz”. Systems under test included BMDP, Data Desk, Excel, GLIM, ISP, SAS, SPSS, S-PLUS, STATGRAPHICS. The results show considerable problems even in basic features of well-known systems. For all our test exercises, the computational solutions are well known. The omissions and failures observed here give some suspicions of what happens in less well-understood problem areas of computational statistics. We cannot take results of data analysis systems at face value, but have to submit them to a large amount of informed inspection. Quality awareness still needs improvement.