Cache evaluation and the impact of workload choice

The selection of the "best" parameters for a cache design, such as size, mapping algorithm, fetch algorithm, line size, etc., is dependent on the expected workload. Similarly, the machine performance is sensitive to the cache performance which itself depends on the workload. Most cache designers have been greatly handicapped in their designs by the lack of realistic cache performance estimates. Published research generally presents data which is unrealistic in some respects, and available traces are often not representative. In this paper, we present measurements from a very wide variety of traces: there are 49 traces, taken from G machine architectures, (370, 360, VAX, MG8000, Z8000, CDC 6400), coded in 7 source languages. Statistics are shown for miss ratios, the effectiveness of prefetching in terms of both miss ratio and its effect on bus traffic, the frequency of writes, reads and instruction fetches, and the frequency of branches. Some general observations are made and a "design estimate" set of miss ratios are proposed. Some "fudge" factors are proposed by which statistics for workloads for one machine architecture can be used to estimate corresponding parameters for another (as yet unrealized) architecture. 1. I n t r o d u c t i o n Almost all medium and high performance machines and most high performance microprocessors now being designed will include cache memories to be used for instructions, for data or for both. There are a number of choices to be made regarding the cache including size, line size (block size), mapping algorithm, replacement algorithm, writeback algorithm, split (instructions/data) vs. unified, fetch algorithm, et cetera; see [Smit82] for a detailed discussion of these issues. Making the "best" choices and selecting the *'best" parameters (with respect to cost and performance) depends greatly on the workload to be expected [Macd84]. For example, a cache which achieves a 99% hit ratio may cost 80% more than one which achieves 98%, may increase the CPU cost by 25°7o and may only boost overall CPU performance by 8%; that suggests that the higher performing cache is not cost effective. However, if the same two designs yield hit ratios of 90°70 and 80~ respectively, and if the performance increase would be 50~, then different conclusions might well be reached. Computer architects have been handicapped by the lack of generally available realistic cache workload estimates. *Tke material prtseated here i* bued on reJe~rck n p p o r t ~ ia part by the Nation*! Science Foundation under g n a t DCR-IU202591 and by tie Ddeue Ad*~nced Rele~rck Project; Agency ¢uder contract N000~9-82-CC,-¢235. Computer time 5 u been provided by tke Stanford Linear Accderatot Center nnder Department of EuerKy contract DEA@0~-?~hSF-00SIS. While there are hundreds of published papers on cache memories (see [Smit82] for a partial bibliography), only a few present usable data. A large fraction contain no measurements at all. Almost all of the papers that do present measurements rely on trace driven simulation using a small set of traces, and for reasons explained further below, those gra.:es are likely to be unrepresentative of the results to be expected in practice. There do exist some realistic numbers, as we note below, but they are hardly enough to constitute a design database. The purpose of this paper is discuss and explain workload selection as it relates to cache memory design, and to present data from which the designer can work. We have used 49 program address traces taken from 0 (or 5, if the 300 and 370 are the same) machine architectures (VAX, 370, 300/91, Z8000, CDC 0400, M08000), derived from 7 programming languages (Fortran, 370 Assembler, APL, C, LISP, AlgolW, Cobol) to compute overall, instruction and data miss ratios and bus traffic rates for various cache designs; these experiments show the variety of workload behavior possible. Characteristics of the traces are tabulated and the effects of some design choices are evaluated. Finally, we present what we consider to be a "re~onable" set of numbers with which we believe designers can comfortably work. In that discussion, we also suggest some "fudge" factors, which indicate how realistic (or available) number~ for machine architecture MI under workload conditions W1 can be used to estimate similar parameters for architecture M2 under workload WI*. In the remainder of this section, we discuss additional background for our measurement results. First we consider the advantages and disadvantages of trace driven simulation. Then we review some {possible) eases of performanee misprediction and also discuss some published and valid miss ratio figures. The second section discusees the traces used. The measurement results and analysis are in section 3, and in section 4 we propose target workload values and factors by which one workload can be used to estimate another. Section 5 summarizes our findings. 1.1 . T r a c e Dr iven Simula t ion A p r o g r a m a d d r e n t race is a trace of the sequence of (virtual) addresses accessed by a computer program or programs. T r a c e dr iven l imul&tlon involves driving a simulation model of a system with a trace of external stimuli rather than with a random number generator. Trace driven simulation is a very good way to study many aspects of cache design and performance, for a number of reasons. First, it is superior to either pure mathematical models or random number driven simulation because there do not currently exist any generally accepted or believable models for those characteristics of program behavior that determine cache performance; thus it is not possible to specify a realistic model nor to drive a simulator with a good representation of a program. A trace properly 0149-7111/85/0000/0064501.00 © 1985 IEEE 64 represents at least one real program, and in certain respects can be expected to drive the simulator correctly. It is important to note that a trace reflects not only the program traced and the functional architecture of the machine (instruction set) but also the design a r c h i t e c t u r e (higher level implementation). In particular, t h e n u m b e r o f m e m o r y references k affected by the width o f the data path to memory : fetching two four-byte instructions requires 4, 2 or 1 memory reference, depending on whether the memory interface is 2, 4 or 8 bytes wide. It also depends on how much "memory" the interface itself has; if one request is for 4 bytes, the next request is for the next four bytes, and the interface is 8 bytes wide, then fewer fetches will result if the interface "remembers" that it has the target four bytes of the second fetch rather than redoing the fetch. The interface can be quite complex, as with the lfetch buffer in the VAX 11/780 [Clar83] and can behave differently for instructions and data. (A trace should reflect, to the greatest possible extent, only the functional architecture; the design architecture should and usually can be emulated in the simulator.) A simulator is also much better in many ways than the construction of prototype designs. It is far faster to build a simulator, and the design being simulated can be varied easily, sometimes by just changing an input parameter. Conversely, a hardware prototype can require man-years to build and can be varied little if at all. Also, the results of a live workload tend to yield slightly different results (e.g. 1% to 3%) from run to run, depending on the random setting of initial conditions such as the angular position of the disks [Curt75]. For the reasons given above, trace driven simulation has been used for almost every research paper which presents cache measurements, with a few exceptions discussed below. There are, however, several reasons why the results of trace driven simulations should be taken with a grain of salt. (1) A trace driven simulation of a million memory addresses, which is fairly long, represents about 1/30 of a second for a machine such as the IBM 3081, and only about one second for an M68000; thus a trace is only a very small sample of a real workload. (2) Traces seldom are taken from the "messiest" parts of large programs; more often they are traces of the initial portions of small programs. (3) It is very difficult to trace the operating system (OS) and few OS traces are available. On many machines, however, the OS dominates the workload. (4) Most real machines task switch every few thousand instructions and are constantly taking interrupts. It is difficult to include this effect accurately in a trace driven simulation and many simulators don't try. (5) The sequence of memory addresses presented to the cache can vary with hardware buffers such ~ prefetch buffers and loop buffers, and is certainly sensitive to the data path width. Thus the trace itself may not be completely accurate with ree.pect to the implementation of the architecture. (0) In running machines, a certain (usually small) fraction of the cache activity is due to input/output; this effect is seldom included in trace driven simulations. In this paper we are primarily concerned with items 1-3 immediately above. By presenting the results of a very large number of simulations, one can get an idea of the range of program behavior. Included are two traces of IBM's MVS operating system, which should have performance that is close to the worst likely to be observed. 1.2. Rea l W o r k l o a d s and Ques t ionab le Estlmattm There arc only a small number of papers in which provide measurements taken by hardware monitors from running machines. In [Mila75] it is reported that a 16K cache on an IBM 370/105-2 running VS2 had a 0.94 hit ratio, with 1.6 fetches per instruction and .22 stores/instruction; it is also found that 73% of the CPU cycles were used in supervisor state. Merrill [Merr74] found cache hit ratios for a 16K cache in the 370/168 of 0.932 to 0.997 for six applications programs, and also reports that the performance (MI