Analysis of Human Factors Engineering Experiments: Characteristics, Results, and Applications

Abstract : Two hundred thirty-nine experiments published in the journal 'Human Factors' during the period from 1958 to 1972 were analyzed for the purpose of discovering the characteristics of their experimental plans, the quality and character of their results, and the degree to which these results had been applied to real systems. The analysis revealed that these experiments investigated too small an experimental space, showed essentially no diversity in their selection of a basic experimental design, collected far more redundant data than was needed, and failed to properly handle the irrelevant variance arising from sequence effects. When a survey was made of those who conducted the experiments, it was discovered that slightly more than half of the experiments had been done to find answers of general applicability; less than a third were known or believed to have influenced the design of a real system. A majority of the investigators said they would not do their experiments any differently if they were to repeat them today. Some limited discussion on the implications of this analysis for an improved experimental methodology is included.