MARS: A multi-level array representation for simulation data

Abstract In the numerical simulation domain, owing to the huge size of data and the complexity of implementing the domain specific applications, a database-centric approach for handling multidimensional simulation data is gaining considerable attention. Array databases provide an optimized set of features to support administrating multidimensional data; representing simulation data with an array can be an optimal choice. Generally, query performance on sparsely filled arrays, especially when empty cells are placed between adjacent elements, can be poor. In this context, previous studies focused on the compact representation of simulation data by reducing the number of empty cells between adjacent elements as possible. However, these methods inevitably lose the original spatial structure of elements (i.e., the relative distance and direction among elements), making it impossible to utilize the built-in multidimensional operators provided by array databases. In this paper, we propose MARS, a multi-level array representation for simulation data. MARS utilizes multiple level arrays with various resolutions to cope with the two addressed problems. In the MARS representation, elements tend to be concentrated into dense array regions, where each region is selectively stored in one of the level arrays that most reduces the empty cells between adjacent elements. Unlike existing methods, MARS retains the spatial structure of elements, and thus no additional efforts to reorganize the original spatial structure for query processing is required. We built MARS on top of SciDB and implemented a specialized command line tool for MARS. We present methods and optimized operators for query processing over MARS. We evaluate the performance of MARS using two real-world numerical simulation datasets.