An experimental study of priority queues in external memory

A priority queue is a data structure that stores a set of items, each one consisting of a tuple which contains some (satellite) information plus a priority value (also called key) drawn from a totally ordered universe. A priority queue supports the following operations on the processed set: access_minimum (returns the item in the set having minimum key), delete_min (returns and deletes the item in the set having the minimum key) and insert (inserts a new item into the set). Priority queues (hereafter PQs) have numerous important applications: combinatorial optimization (e.g. Dijkstra’s shortest path algorithm [7]), time forward processing [5], job scheduling, event simulation and online sorting, just to cite a few. Many PQ implementations currently exist for small data sets fitting into the internal memory of the computer, e.g. k—ary heaps [23], Fibonacci heaps [10], radix heaps [1], and some of them are also publicly available to the programmers (see e.g. the LEDA library [15]). However, in large-scale event simulations or on instances of very large graph problems (as they recently occur in e.g. geographical information systems), the performance of these internal-memory PQs may significantly deteriorate, thus being a bottleneck for the overall application. In fact, as soon as parts of the PQs do not fit entirely into the internal memory of the computer, but reside in its external memory (e.g. in the hard disk), we may observe a heavy paging activity of the external-memory devices because the pattern of memory accesses is not tuned to exhibit any locality of reference. Due to the technological features of current disk systems [17], this situation may determine a slow down of 5 or 6 orders of magnitude in the final performance of each PQ-operation 1. Consequently, it is required to design PQs which take explicitly into account the physical properties of the disk systems in order to achieve efficient I/O-performances that allow these data structures to be plugged successfully in software libraries.