Software caching vs. prefetching

The performance gap between memory subsystem and high-performance processors is ever-increasing. Prefetching is one method to bridge this performance gap. Prefetching has been proposed for array-based and pointer applications, typically using software-based techniques with the help of the compiler. Prefetching suffers from certain disadvantages such as an increase in memory traffic, an increase in the number of executed instructions, and an increase in memory requirement for some cases.In this paper, we investigate the technique of software caching for applications that perform searches or sorted insertions. For data structures larger than the processor data cache, such a search or sorted insert may result in multiple cache misses before the correct value is found. In a software caching approach, a small software buffer is maintained that records the most recently added values (these values are the ones that are later searched) along with their addresses and is consulted during a search or an insert. In this paper, we present results of our initial experiments. We found that for applications involving a search, software caching performs as high as 30% better than the original application. Also, this technique executes upto 14% less instructions and the number of cache accesses decrease by around upto 18%. We also compared this technique against software prefetching and found that in some cases, the performance improvement was as high as 40%. This technique also has an added advantage of using less memory space than prefetching (all the improvements were obtained for very small software caches). These initial results encourage a more in-depth study of this technique.