Experimental evaluation of software documentation formats

Abstract Four controlled experiments investigated the effectiveness of different documentation formats for presenting information about computer programs. Nine different documentation formats (some quite novel) were created by varying three different forms of symbology (natural language, constrained language, and ideograms) and three different spatial arrangements (sequential, branching, and hierarchical). Professional programmers used these formats as aids to comprehend (Experiment 1), code (Experiment 2), debug (Experiment 3), and modify (Experiment 4) modular-sized programs. For each programming task, a model of the cognitive activities a programmer would perform generated hypotheses about the effectiveness of different formats. Natural language was found to be less effective in assisting most of the tasks studied than a constrained language or ideograms. A smaller effect was observed (less frequently than expected) for the spatial arrangment in situations where control flow information aided the task. The largest effect in the experiments, individual differences among the participants, accounted for between a third and one-half of the variation in performance.

[1]  Thomas R. G. Green,et al.  Scope Marking in Computer Conditionals - A Psychological Evaluation , 1977, Int. J. Man Mach. Stud..

[2]  Harlan D. Mills,et al.  Understanding and Documenting Programs , 1982, IEEE Transactions on Software Engineering.

[3]  Stephen H. Caine,et al.  PDL: a tool for software design , 1975, AFIPS '75.

[4]  B. Adelson.  Problem solving and the development of abstract categories in programming languages , 1981, Memory & cognition.

[5]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[6]  John W. Bailey,et al.  An empirical evaluation of software documentation formats , 1982, CHI '82.

[7]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[8]  Deborah A. Boehm-Davis,et al.  Documentation of Concurrent Programs , 1985 .

[9]  B. A. Sheil,et al.  The Psychological Study of Programming , 1981, CSUR.

[10]  Ruven E. Brooks,et al.  Towards a Theory of the Comprehension of Computer Programs , 1983, Int. J. Man Mach. Stud..

[11]  Ron Weber,et al.  Conditional Statements and Program Coding: An Experimental Evaluation , 1984, Int. J. Man Mach. Stud..

[12]  A. Paivio.  Imagery and verbal processes , 1972 .

[13]  Edsger W. Dijkstra,et al.  Notes on structured programming , 1970 .

[14]  J. B. Brooke,et al.  Experimental studies of flowchart use at different stages of program debugging , 1980 .

[15]  Ira R. Forman,et al.  Psychological perspectives for software science , 1984, Inf. Process. Manag..

[16]  Susan Wiedenbeck,et al.  Beacons in Computer Program Comprehension , 1986, Int. J. Man Mach. Stud..

[17]  Harry Katzan.  Systems design and documentation : an introduction to the HIPO method , 1976 .

[18]  Robert L. Sedlmeyer,et al.  Knowledge-based fault localization in debugging: preliminary draft , 1983 .

[19]  Ron Weber,et al.  Structured tools and conditional logic: an empirical investigation , 1986, CACM.

[20]  Thomas R. G. Green,et al.  Pictures of programs and other processes, or how to do things with lines , 1982 .

[21]  Ian Barrodale,et al.  Elementary computer applications in science, engineering, and business , 1971 .

[22]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[23]  Andrew Ortony,et al.  Remembering, Understanding, and Representation* , 1978 .

[24]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[25]  Stephen S. Yau,et al.  Design Stability Measures for Software Maintenance , 1985, IEEE Transactions on Software Engineering.

[26]  Fred P. Brooks,et al.  The Mythical Man-Month , 1975, Reliable Software.

[27]  Michael E. Atwood,et al.  Flowcharts vs. Program Design Languages: An Experimental Comparison , 1978 .

[28]  J. Reitman,et al.  Knowledge organization and skill differences in computer programmers , 1981, Cognitive Psychology.

[29]  B. Curtis,et al.  Substantiating programmer variability , 1981, Proceedings of the IEEE.

[30]  Robert L. Sedlmeyer,et al.  Knowledge-based fault localization in debugging , 1983, J. Syst. Softw..

[31]  Kate Ehrlich,et al.  Empirical Studies of Programming Knowledge , 1984, IEEE Transactions on Software Engineering.

[32]  Thomas R. G. Green,et al.  When do diagrams make good computer languages , 1979 .

[33]  Ben Shneiderman,et al.  Experimental investigations of the utility of detailed flowcharts in programming , 1977, CACM.

[34]  T. R. G. Green.  Conditional program statements and their comprehensibility to professional programmers , 1977 .

[35]  R. Kirk.  Experimental Design: Procedures for the Behavioral Sciences , 1970 .

[36]  Ben Shneiderman,et al.  Software psychology: Human factors in computer and information systems (Winthrop computer systems series) , 1980 .

[37]  Ruven E. Brooks,et al.  Studying programmer behavior experimentally: the problems of proper methodology , 1980, CACM.

[38]  Elliot Soloway,et al.  A Goal/Plan Analysis of Buggy Pascal Programs , 1985, Hum. Comput. Interact..

[39]  Bill Curtis,et al.  Third time charm: Stronger prediction of programmer performance by software complexity metrics , 1979, ICSE 1979.

[40]  Richard Kammann,et al.  The Comprehensibility of Printed Instructions and the Flowchart Alternative , 1975 .

[41]  J. B. Brooke,et al.  An experimental study of flowcharts as an aid to identification of procedural faults , 1980 .

[42]  B. Curtis,et al.  Measurement and experimentation in software engineering , 1980, Proceedings of the IEEE.

[43]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[44]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[45]  Harlan D. Mills,et al.  Structured programming - theory and practice , 1979, The systems programming series.

[46]  P. Wright,et al.  Written information: Some alternatives to prose for expressing the outcomes of complex contingencies. , 1973 .

[47]  Alfred V. Aho,et al.  Data Structures and Algorithms , 1983 .

[48]  Mark Weiser,et al.  Programmers use slices when debugging , 1982, CACM.

[49]  John B. Black,et al.  Software psychology: The need for an interdisciplinary program , 1986, Proceedings of the IEEE.

[50]  R E Mayer.  Comprehension as affected by structure of problem representation , 1976, Memory & cognition.

[51]  Michael E. Atwood,et al.  Flowcharts versus program design languages: an experimental comparison , 1983, CACM.

[52]  Michael E. Atwood,et al.  Cognitive structures in the comprehension and memory of computer programs: an investigation of compu , 1978 .

[53]  Edsger W. Dijkstra,et al.  Structured programming , 1972, A.P.I.C. Studies in data processing.

[54]  Arthur S. Blaiwes,et al.  Formats for presenting procedural instructions. , 1974 .

[55]  Bill Curtis,et al.  Modern Coding Practices and Programmer Performance , 1979, Computer.

[56]  David J. Gilmore,et al.  An Investigation of the Utility of Flowcharts During Computer Program Debugging , 1984, Int. J. Man Mach. Stud..

[57]  J. Elashoff,et al.  Multiple Regression in Behavioral Research. , 1975 .

[58]  Kate Ehrlich,et al.  Cognitive strategies and looping constructs: an empirical study , 1983, CACM.

[59]  Iris Vessey,et al.  Expertise in Debugging Computer Programs: A Process Analysis , 1984, Int. J. Man Mach. Stud..

[60]  Ben Shneiderman,et al.  Control flow and data structure documentation: two experiments , 1982, CACM.