Performance evaluation: Experimental computer science at its best

What is experimental computer science? This question has been widely discussed ever since the Feldman Report was published (1979 [18]). Many computer scientists believe that survival of their discipline is intimately linked to their ability to rejuvenate experimentation. The National Science Foundation instituted the Coordinated Experimental Research Program (CERP) in 1979 to help universities set up facilities capable of supporting experimental research. Other agencies of government are considering similar programs. Some industrial firms are offering similar help through modest cash grants and equipment discounts. What is experimental computer science? Surprisingly, computer scientists disagree on the answer. A few believe that computer science is in flux—making a transition from theoretical to experimental science—and, hence, no operational definition is yet available. Some believe that it is all the non-theoretical activities of computer science, especially those conferring “hands-on” experience. Quite a few believe that it is large system development projects—i.e., computer and software engineering—and they cite MIT's Multics, Berkeley's version of Bell Labs' UNIX, the ARPAnet, IBM's database System R, and Xerox's Ethernet-based personal computer network as examples. These beliefs are wrong. There are well-established standards for experimental science. The field of performance evaluation meets these standards and provides examples of experimental science for the rest of the computing field.

[1]  K. Mani Chandy,et al.  Open, Closed, and Mixed Networks of Queues with Different Classes of Customers , 1975, JACM.

[2]  D. Sayre,et al.  Is automatic “folding” of programs efficient enough to displace manual? , 1969, CACM.

[3]  Martin Reiser,et al.  Mean-value analysis and convolution method for queue-dependent servers in closed queueing networks , 1981, Perform. Evaluation.

[4]  Peter J. Denning,et al.  The Operational Analysis of Queueing Network Models , 1978, CSUR.

[5]  K. Mani Chandy,et al.  Computational algorithms for product form queueing networks , 1980 .

[6]  Irving L. Traiger,et al.  Evaluation Techniques for Storage Hierarchies , 1970, IBM Syst. J..

[7]  Barbara S. Brawn,et al.  Sorting in a paging environment , 1970, Commun. ACM.

[8]  Jeffrey Buzen,et al.  Analysis of system bottlenecks using a queueing network model , 1971, SIGOPS Workshop on System Performance Evaluation.

[9]  Richard J. Lipton,et al.  Social processes and proofs of theorems and programs , 1977, POPL.

[10]  Robert W. O'Neill,et al.  Experience using a time-shared multi-programming system with dynamic address relocation hardware , 1967, AFIPS '67 (Spring).

[11]  Yonathan Bard,et al.  Some Extensions to Multiclass Queueing Network Analysis , 1979, Performance.

[12]  Jacob Feldman,et al.  Rejuvenating experimental computer science , 1979 .

[13]  Brian Randell,et al.  A note on storage fragmentation and program segmentation , 1969, CACM.

[14]  Jeffrey P. Buzen,et al.  Computational algorithms for closed queueing networks with exponential servers , 1973, Commun. ACM.

[15]  Peter J. Denning,et al.  Working Sets Past and Present , 1980, IEEE Transactions on Software Engineering.

[16]  Laszlo A. Belady,et al.  An anomaly in space-time characteristics of certain programs running in a paging machine , 1969, CACM.

[17]  Laszlo A. Belady,et al.  Dynamic space-sharing in computer systems , 1969, Commun. ACM.

[18]  Laszlo A. Belady,et al.  A Study of Replacement Algorithms for Virtual-Storage Computer , 1966, IBM Syst. J..

[19]  Barbara S. Brawn,et al.  Program behavior in a paging environment , 1899, AFIPS '68 (Fall, part II).

[20]  Jeffrey P. Buzen,et al.  Queueing Network Models of Multiprogramming , 1971, Outstanding Dissertations in the Computer Sciences.

[21]  Hisashi Kobayashi,et al.  Queuing Networks with Multiple Closed Chains: Theory and Computational Algorithms , 1975, IBM J. Res. Dev..

[22]  Stephen S. Lavenberg,et al.  Mean-Value Analysis of Closed Multichain Queuing Networks , 1980, JACM.

[23]  Jerome A. Feldman,et al.  Rejuvenating experimental computer science: a report to the National Science Foundation and others , 1979, CACM.

[24]  Peter J. Denning,et al.  Measuring and Calculating Queue Length Distributions , 1980, Computer.