Can You Trust Your Memory Trace? A Comparison of Memory Traces from Binary Instrumentation and Simulation

Simulation is employed extensively to perform exploration of design spaces by computer designers. Contemporary simulation environments are now increasingly complex comprising of support for multiple cores and full operating systems. Resource use between simulation environments vary widely because of these different system contexts and the fact that multi-threaded applications have intrinsic non-determinism. In addition, more recent simulation environments use Dynamic Binary Instrumentation (DBI) traces collected on the system context (OS, library, threading API) of the host system. Methodologies that have been employed to validate and compare simulation frameworks are usually limited to comparing CPI and cache statistics and do not provide a detailed function-level breakdown or understanding of the source of mismatches. In this work, we attempt to identify and quantify the true sources of mismatch between a DBI framework and a full system simulation framework. We use memory traces of multithreaded applications that have been annotated with function call information to allow for a breakdown of the source of mismatch within an application. To the best of our knowledge, this level of detail in comparison has not been attempted before, especially with traces of multi-threaded applications. In this study, we find that the sources of mismatch come mainly from threading mechanisms/threading API function calls, Library/System function calls and User Space condition synchronization. Based on the results of the study, we identify specific functions in each category of mismatch. We then propose a few ways to close the gap and enable more reliable simulation for design space exploration.