Handling Bias from Individual Differences in between-Subject Holistic Experimental Designs.

Abstract : The effects of expected biases from individual differences are quantified in this report. Techniques for handling them are discussed, particularly in the context of 2 k-p holistic experiments. Advantages of a holistic approach for equipment design research are presented. In contrast with the traditional few-factors-at-a-time approach, this approach has its fundamental philosophy the need to investigate all of the potentially critical factors in the same experiment. A sequential strategy and bundle of techniques make the approach feasible. To follow this philosophy increases the generalizability of the results and reduces the dangers of misinterpretation when critical factors are held constant. Data collection costs are also reduced over what they would be for equivalent information collected a few factors at a time. To make this multifactor approach economical, the experiment is run initially using only one subject per experimental condition. The obvious confounding of the subject and configuration effects in the performance scores creates a concern regarding biased results and the risk of misinterpreting them. Computer simulations and analyses were performed to better understand the anatomy of the subject-related bias problem and to evaluate ways to reduce it.