The J2EE ECperf benchmark results: transient trophies or technology treasures?

ECperf, the widely recognized industry standard J2EE benchmark, has attracted a large number of results submissions and their subsequent publications. However, ECperf places little restriction on the hardware platforms, operating systems and databases utilized in the benchmarking process. This, combined with the existence of only two primary metrics, makes it difficult to accurately compare the performance of the Application Server products themselves. By mining the full‐disclosure archives for trends and correlations, we have discovered that J2EE technology is very scalable both in a scale‐up and scale‐out manner. Other observed trends include a linear correlation between middle‐tier total processing power and throughput, as well as between J2EE Application Server license costs and throughput. However, the results clearly indicate that there is an increasing cost per user with increasing capacity systems and scale‐up is proportionately more expensive than scale‐out. Finally, the correlation between middle‐tier processing power and throughput, combined with results obtained from a different ‘lighter‐weight’ benchmark, facilitates an estimate of throughput for different types of J2EE applications. Copyright © 2004 John Wiley & Sons, Ltd.