Architectural Simulators Considered Harmful

Much as Edgar Dijkstra in 1968 observed the dangers of relying on the "go to" statement, the authors of this article observe the detrimental effect of overreliance on quantitative simulators. Over time, simulator tools have become more complex and sophisticated. While this seems beneficial, new generations of simulators are built with easily abused modeling abstractions, some with poorly modeled first-order phenomenon. Users treat simulators as black boxes, ignoring their potential errors and relying on validation to irrelevant design points. The lack of documentation and specification makes finding or even being aware of errors challenging. Simultaneously, reviewers demand that researchers use simulation in domains where better approaches exist. The authors elucidate the problems by describing three broad pitfalls of simulators and simulator use, considering examples from modern tools, and they discuss how to avoid each problem. Finally, they propose one consideration for recalibrating evaluation standards: the footprint, which is the breadth of architectural layers that a technique affects.