Should Computer Scientists Experiment More?

Computer scientists and practitioners defend their lack of experimentation with a wide range of arguments. Some arguments suggest that experimentation is inappropriate, too difficult, useless, and even harmful. This article discusses several such arguments to illustrate the importance of experimentation for computer science. It considers how the software industry is beginning to value experiments, because results may give a company a three- to five-year lead over the competition.

[1]  Trevor N. Mudge.  Report on the panel: “how can computer architecture researchers avoid becoming the society for irreproducible results?” , 1996, CARN.

[2]  Les Hatton,et al.  Reexamining the Fault Density-Component Size Connection , 1997, IEEE Softw..

[3]  Nancy G. Leveson,et al.  An experimental evaluation of the assumption of independence in multiversion programming , 1986, IEEE Transactions on Software Engineering.

[4]  Walter F. Tichy,et al.  A Controlled Experiment to Assess the Benefits of Procedure Argument Type Checking , 1998, IEEE Trans. Software Eng..

[5]  Victor R. Basili,et al.  Software errors and complexity: an empirical investigation0 , 1984, CACM.

[6]  Anthony Ralston,et al.  Encyclopedia of Computer Science , 1971 .

[7]  Walter F. Tichy,et al.  An experiment to assess the benefits of inter-module type checking , 1996, Proceedings of the 3rd International Software Metrics Symposium.

[8]  E. Wilson,et al.  The Diversity of Life , 1993, Politics and the Life Sciences.

[9]  Juris Hartmanis,et al.  Turing Award lecture on computational complexity and the nature of computer science , 1994, CACM.

[10]  Edward A. Feigenbaum.  How the “what” becomes the “how” , 1996, CACM.

[11]  Frederick P. Brooks,et al.  The computer scientist as toolsmith II , 1996, CACM.

[12]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[13]  S. L. Pfleeger.  Workshop Defines Problems with Software-Engineering Data , 1996 .

[14]  Raj Reddy.  To dream the possible dream , 1996, CACM.

[15]  Paul Lukowicz,et al.  Experimental evaluation in computer science: A quantitative study , 1995, J. Syst. Softw..

[16]  Adam A. Porter,et al.  Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies , 1997, IEEE Trans. Software Eng..