The Use and Abuse of SPEC: An ISCA Panel

The 30th International Symposium on Computer Architecture featured an evening panel and dinner cosponsored by Intel and IEEE Micro. To kick off this panel, Daniel Citron presented " MisSPECulation: Partial and Misleading Use of SPEC CPU2000 in Computer Architecture Conference " (Proc. 30th Ann. Int'l Symposium Computer Architecture, IEEE CS Press, 2003, pp. 52-59), his position paper from this year's ISCA proceedings. The ISCA program committee and IEEE Micro thank moderator John Hennessy, and pan-elists Daniel Citron, Dave Patterson, and Guri Sohi for providing a lively and thought-provoking discussion.—Pradip Bose, editor-in-chief Hennessy: We wanted to talk about some of the ways in which performance numbers are gathered and published in papers. We're going to start with a presentation by each of the panel members. We allocated different amounts of time, according to how much they have to say that's useful. First, Daniel Citron will come up; he's from IBM Israel, and he's going to talk about his work, which you've probably seen in the proceedings. Citron: OK, let's look at the misleading use of SPEC. Let's start with the scene. Many of you recognize the various places that SPEC use/mis-use takes place—recent computer architecture conferences. I started to look at the ISCA, Micro, and HPCA conferences of the past three years. Each published a varying number of papers, and SPEC use was widespread: 209 papers were published and 66 percent of them used SPEC. Research for the earliest conference , ISCA 2001, was probably done by the end of 2000. But that was a year after SPEC announced CPU2000, so you had plenty of time to get the new benchmarks and use them. Next, the victim: SPEC CPU. It's currently in its fourth version. SPEC CPU2000 is CPU intensive, and it measures the performance of the processor, memory, and compiler. And if you look at the paper breakdown [Figure 1], you can see that data path and memory papers are predominant in these proceedings , and they use SPEC; up to 90 percent of the papers that give values use SPEC. OK, so what's the crime? You know what, I'll plead it down to a misdemeanor: the partial use of CPU2000.