Large-scale lossy data compression based on an a priori error estimator in a Spectral Element Code

An increasing bottleneck for high performance computing and large-scale simulations is the I/O of supercomputers. We present a highly scalable lossy data compression algorithm with controllable error based on an a priori estimator, applied to complex curvilinear meshes. Integral transforms, such as, the the discrete Chebyshev transform (DCT) or discrete Legendre transform (DLT) due to their energy compactness property can compress a signal optimally. In these approaches the data field is mapped to spectral space and subsequently truncated within a user-specified tolerance which ensures that the recovered signal is below a desired error threshold. The truncated data field is compressed for high I/O speed and low storage using bitwise encoding. The method presented here is endowed with an a priori error estimator to allow for a dynamic compression ratio which can be as low as 3% compression for data visualization even in cases as complex as fully developed turbulent flow. For checkpointing problems however, the compression ratio is problem dependent. We also derived an orthogonal transform compatible with the spectral discretization on GaussLegendre-Lobatto curvilinear grids, which displays superior performance to the traditional DCT. The algorithm is implemented in the spectral-element code Nek5000 and tested on highly turbulent large-scale simulation data of up to 3.2 billion degrees of freedom. The implementation via tensor products is highly efficient, reducing the flops in matrix-vector multiplications from O(N2d) down to O(Nd+1) in a d-dimensional space.