Your favorite simulator here " Considered Harmful

ion errors than an unknown black box. Reviewers and the community needs to change its mindset as well – having blind faith in “standard tools,” while completely discounting other tools is not appropriate. We revisit the issue of open versus in-house tools in Section 4. 3.2. Pitfall 2: False confidence from validation overgeneralization in simulator papers, or tool misuses Simulator writers typically make narrow and factually consistent statements about validation, and some examples are below. However, the nature of validation is often misunderstood by users, and these tools are put to use in ways not intended for, including making quantitative generalizations. gem5’s OOO model is widely used, but as observed in a recent paper [13] and our observations above, it has several specification errors. Though the gem5 authors themselves do not claim it as such, some do claim it is a “validated simulator.” Clearly, this cannot be taken as all effects modeled. For instance, a technique that works on the instruction frontend must pay attention to gem5’s baseline and first fix the specification error described here [13]. Considering McPAT, according to their own documentation and code comments, constants are sometimes chosen to match the validation targets. We agree this is a reasonable decision in some cases, especially when highly customized logic is employed (e.g. functional unit implementations). The danger is when researchers attempt to generalize the results outside the validated processors. These constants will likely not be appropriate. For GPUWattch, it might be tempting for researchers to perform sensitivity studies by varying McPAT parameters. The path of least resistance would be to use the same scaling factors, instead of measuring the power of a known GPU and deriving new scaling factors using the GPUWattch methodology. For reasons described in the previous section, we argue that without obtaining new scaling factors, this type of sensitivity analysis would be inappropriate. Suggestions: Use with caution validated simulators. Look for details on the simulator’s design and factor those decisions