A methodology for evaluating time-shared computer system usage

The development of time-shared computer systems has led to major technical and philosophical changes in the computer field in this decade. A large number of designers, manufacturers, and users of such systems have expended great amounts of effort in the development of the capabilities of the computer and the means to use it. However, little or no effort has yet been expended to evaluate these systems in terms of their usefulness for present or future customers. The research reported here has focused on the development of a methodology through which time-shared computer system usage can be evaluated. It is based on a study of the characteristics and design of present and proposed computer systems, as well as relevant behavioral theory and research. Five categories of variables are included in the resulting methodology, namely those which are measures of* (1) the cost of using the system? (2) the performance produced through the use of the computer system; (3) the speed with which results could be produced; (4) the amount of learning resulting from the use of the computer system; and (5) the attitudes of the users of the computer system. The methodology developed was tested experimentally through evaluating usage of two computer systems, each exhibiting certain characteristics of both time-sharing and batch-processing. The primary problem under study was the effect of rapid feedback and unlimited computer access in a problem-solving situation — the secondary investigation involved the effect of qualitatively different feedback upon computer programming. The testing of the methodology indicated that the following are important parameters in an evaluation: (1) the nature of the application; (2) the interaction between the relative accessibility of the computer system studied and the length of time available for the user to perform the required task; and (3) the monetary value assigned to both the man^ and the computer system^ time. As a result of the two evaluations attempted, there appears to be sufficient evidence for concluding that time-shared computer usage can be evaluated and that the methodology presented may be both sufficiently effective and general to be used for evaluation of time-sharing usage over a wide range of applications. However, no attempt should be made to generalize the results of these specific evaluations to all timesharing systems or applications. The testing of the methodology yielded unexpected findings related to the effect of continuous sessions with the time-shared computer system. Measurement of the changes in user performance indicated that the first effective interaction of a session with the time-shared computer resulted in a greater amount of problem-solving than subsequent interactions during that session — a result in conflict with current literature. The report concludes with a discussion of the implications for the role of feedback delays in a man-machine system.

[1]  L. Festinger,et al.  Social pressures in informal groups , 1950 .

[2]  G. Homans.  The human group , 1952 .

[3]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences. , 1957 .

[4]  Christopher Strachey,et al.  Time sharing in large, fast computers , 1959, IFIP Congress.

[5]  Galanter,et al.  Automatic Teaching The State Of The Art , 1959 .

[6]  W. C. Christian,et al.  Management Organization and the Computer , 1961 .

[7]  Arthur M. Cohen,et al.  CHANGING SMALL GROUP COMMUNICATION NETWORKS , 1961 .

[8]  Robert C. Daley,et al.  An experimental time-sharing system , 1962, AIEE-IRE '62 (Spring).

[9]  Alexander L. Pugh,et al.  Dynamo user's manual , 1963 .

[10]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.

[11]  Robert Stotz.  Man-machine console facilities for computer-aided design , 1963, AFIPS '63 (Spring).

[12]  David M. Messick,et al.  Computer-controlled experiments in psychology. , 1964, Behavioral science.

[13]  William Herbert Desmonde,et al.  Real-time data processing systems : introductory concepts , 1964 .

[14]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1964 .

[15]  Ivan E. Sutherland,et al.  Sketch pad a man-machine graphical communication system , 1964, DAC.

[16]  J. B. Dennis.  PROGRAM STRUCTURE IN A MULTI-ACCESS COMPUTER , 1964 .

[17]  B. Berelson,et al.  Human Behavior: An Inventory of Scientific Findings , 1964 .

[18]  R. M. Fano,et al.  Some thoughts about the social implications of accessible computing , 1899, AFIPS '65 (Fall, part I).

[19]  A. L. Scherr,et al.  AN ANALYSIS OF TIME-SHARED COMPUTER SYSTEMS , 1965 .

[20]  K R Hammond,et al.  Differential feedback in two multiple-cue probability learning tasks. , 1965, Behavioral science.

[21]  G F Weinwurm.  DATA ELEMENTS FOR A COST REPORTING SYSTEM FOR COMPUTER PROGRAM DEVELOPMENT , 1966 .

[22]  W. J. Erikson,et al.  A PILOT STUDY OF INTERACTIVE VERSUS NONINTERACTIVE DEBUGGING , 1966 .

[23]  Dick Schutz,et al.  The Effect of Educational Research on Classroom Instruction , 1966 .

[24]  Donald C. Carroll.  Implications of on-line, real-time systems for managerial decision making. , 1966 .

[25]  J A Swets,et al.  Computer-aided instruction in perceptual identification. , 1966, Behavioral science.

[26]  R. M. Fano,et al.  Time-Sharing on Computers , 1966 .

[27]  D. C. Phillips,et al.  Programming Real-Time Computer Systems , 1966 .

[28]  Bernard Berelson,et al.  Human behavior: An inventory of scientific findings. , 1967 .

[29]  James H. Morris,et al.  On-Line Computation and Simulation: The OPS-3 System. , 1967 .

[30]  Malcolm C. Harrison,et al.  On-line computation and simulation : the OPS-3 system , 1967 .

[31]  Peter Calingaert,et al.  System performance evaluation , 1967, Commun. ACM.

[32]  J. R. King,et al.  The Challenge of the Computer Utility , 1967 .

[33]  E. Mumford.  Computers and the world of the future. , 1968, British dental journal.

[34]  Maurice V. Wilkes,et al.  Time-sharing computer systems , 1968 .

[35]  E. E. Grant,et al.  Exploratory experimental studies comparing online and offline programming performance , 1968, CACM.