New data compression algorithms for space applications: software and hardware implementations

Space missions comprise a large set of different strategies, instruments and scientific objectives. The volume of data to be transmitted to ground increases as the quality and complexity of the instruments increase. Considering the constraints of the space environment, a direct consequence of this is the need of algorithms to compress the data. Universal coders offering high compression ratios obviously benefit the vast majority of space missions. Gaia is a very challenging mission in terms of data compression. Initially, we improved an existing compression system which was based on two stages, namely, a pre-processing stage followed by a coder. Our improvement led to adequate compression results. However, when considering the latest mission design we realized that a deep revision was required, especially in the coder stage. We developed a set of analysis tools to study the input data and evaluate the compression design. It made possible the definition of a pre-processing stage sensibly improving the final compression ratio. Regarding the coder, we realized that the 121.0 recommendation of the Consultative Committee for Space Data Systems (CCSDS) was too complex to be used in Gaia. A solution was to use its core alone (the Rice coder), previously calibrated for each data field, removing the adaptive structure. However, this system was too sensitive to outliers, so we proposed a modification that successfully solved this problem. Not only this, but we also designed a new entropy coder (PEC, Prediction Error Coder), specifically designed for the optimum coding of prediction errors with significant amounts of outliers and noise. Using a theoretical analysis we have found that the performance of PEC is quite satisfactory. In particular, we have found that PEC is extremely fast and efficient, thus meeting the specifications of Gaia. PEC must be calibrated to each kind of data. Despite of its robustness in front of variations, an adaptive stage (similar to that of CCSDS 121.0) would make it really applicable to most missions. The answer to this challenge is the Fully-Adaptive PEC coder (FAPEC). Its operation is quite simple. FAPEC builds a logarithmic-like histogram to optimize the resources of the system. Once a data block is input, the coder analyses the histogram, checking some probability thresholds, and choosing a nearly-optimal configuration for PEC. We have validated FAPEC using synthetically generated data following distributions typically obtained after a pre-processing stage. We considered extremely important to verify that both PEC and FAPEC work properly when realistic data are to be compressed. To achieve this, we built a corpus of data files made of representative space-based data, obtained from very different instruments. Tests with these data have demonstrated that FAPEC outperforms CCSDS 121.0 in many cases. Finally, we have developed a version of FAPEC in hardware more specifically, in an FPGA board. This kind of solutions result in low power consumption combined with high throughputs. Some aspects of the FAPEC algorithm had to be modified to ease the implementation. Once this was done, we validated with simulations the correct operation of the hardware implementation of FAPEC. The results obtained using realistic data from the Gaia mission were excellent, obtaining improvements of up to 10% with respect to CCSDS 121.0. Moreover, the latency and the resource allocation were very low. In conclusion, we have developed PEC, an efficient and quick entropy coder that is an excellent option for scenarios with low variations in the statistics. If an adaptive algorithm is required, FAPEC is the best option. It produces the same results 'or even better ' than PEC, without the need of a prior calibration. Its excellent results have convinced us to patent it. Finally, the hardware version has also been developed, obtaining high throughputs with few logical resources.