Dynamic Memory Adjustment for External Mergesort

Sorting is a memory intensive operation whose performance is greatly affected by the amount of memory available as work space. When the input size is unknown or available memory space varies, static memory allocation either wastes memory space or fails to make full use of memory to speed up sorting. This paper presents a method for run-time adjustment of in-memory work space for external mergesort and a policy for allocating memory among concurrent, competing sorts. Experimental results confirm that the new method enables sorts to adapt their memory usage gracefully to the actual input size and available memory space, When multiple sorts compete for memory resources, we found that sort throughput and response time are improved significantly by our policy for memory allocation combined with limiting the number of sorts processed concurrently. Many systems rely on static allocation, that is, work space is allocated when an operation starts and remains unchanged until it finishes. This approach is likely to result in some operations wasting memory while others are starved for memory. There are two reasons why this may happen. First, the input size may be unknown or poorly estimated. Second, in a multiuser environment the workload varies, resulting in varying demands on the available memory. Overall performance can be improved by algorithms that enable operations to adjust their memory usage at run time in response to the actual size of their inputs and fluctuations in total memory demand. Sorting is a frequent operation in database systems. It is used not only to produce sorted output, but also in many sort-based algorithms, such as grouping with aggregation, duplicate removal, sort-merge join and set operations [Gra93]. Sorting can also improve the efficiency of algorithms like nested-loop joins and row retrieval via an index. This paper concentrates on dynamic memory adjustment for sorting but the same approach can be applied to other memory intensive operations.