Optimization Under Uncertainty Versus Algebraic Heuristics: A Research Method for Comparing Computational Design Methods

In this paper, we introduce a research method for comparing computational design methods. This research method addresses the challenge of measuring the difference in performance of different design methods in a way that is fair and unbiased with respect to differences in modeling abstraction, accuracy and uncertainty representation. The method can be used to identify the conditions under which each design method is most beneficial. To illustrate the research method, we compare two design methods for the design of a pressure vessel: 1) an algebraic approach, based on the ASME pressure vessel code, which accounts for uncertainty implicitly through safety factors, and 2) an optimization-based, expected-utility maximization approach which accounts for uncertainty explicitly. The computational experiments initially show that under some conditions the algebraic heuristic surprisingly outperforms the optimization-based approach. Further analysis reveals that an optimization-based approach does perform best as long as the designer applies good judgment during uncertainty elicitation. An ignorant or overly confident designer is better off using safety factors.