Marlin: A High Throughput Variable-to-Fixed Codec Using Plurally Parsable Dictionaries

We present Marlin, a variable-to-fixed (VF) codec optimized for decoding speed. Marlin builds upon a novel way of constructing VF dictionaries that maximizes efficiency for a given dictionary size. On a lossless image coding experiment, Marlin achieves a compression ratio of 1.94 at 2494MiB/s. Marlin is as fast as state-of-the-art high-throughput codecs (e.g., Snappy, 1.24 at 2643MiB/s), and its compression ratio is close to the best entropy codecs (e.g., FiniteStateEntropy, 2.06 at 523MiB/s). Therefore, Marlin enables efficient and high throughput encoding for memoryless sources, which was not possible until now.

[1]  Satoshi Yoshida,et al.  An Efficient Algorithm for Almost Instantaneous VF Code Using Multiplexed Parse Tree , 2010, 2010 Data Compression Conference.

[2]  Abraham Lempel,et al.  A universal algorithm for sequential data compression , 1977, IEEE Trans. Inf. Theory.

[3]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[4]  Serap A. Savari,et al.  Variable-to-fixed length codes and plurally parsable dictionaries , 1999, Proceedings DCC'99 Data Compression Conference (Cat. No. PR00096).

[5]  G. Nigel Martin,et al.  * Range encoding: an algorithm for removing redundancy from a digitised message , 1979 .

[6]  P.G. Howard,et al.  Fast and efficient lossless image compression , 1993, [Proceedings] DCC `93: Data Compression Conference.

[7]  Jyrki Alakuijala,et al.  Gipfeli - High Speed Compression Algorithm , 2012, 2012 Data Compression Conference.

[8]  Danny Harnik,et al.  A Fast Implementation of Deflate , 2014, 2014 Data Compression Conference.

[9]  Hidetoshi Yokoo,et al.  Average-sense optimality and competitive optimality for almost instantaneous VF codes , 2001, IEEE Trans. Inf. Theory.

[10]  Ahmad Al-Rababa'a,et al.  Using bit recycling to reduce the redundancy in plurally parsable dictionaries , 2015, 2015 IEEE 14th Canadian Workshop on Information Theory (CWIT).

[11]  Ross N. Williams,et al.  An extremely fast Ziv-Lempel data compression algorithm , 1991, [1991] Proceedings. Data Compression Conference.

[12]  Jorma Rissanen,et al.  Generalized Kraft Inequality and Arithmetic Coding , 1976, IBM J. Res. Dev..

[13]  Guillermo Sapiro,et al.  The LOCO-I lossless image compression algorithm: principles and standardization into JPEG-LS , 2000, IEEE Trans. Image Process..

[14]  Moncef Gabbouj,et al.  Laplacian modeling of DCT coefficients for real-time encoding , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[15]  David A. Huffman,et al.  A method for the construction of minimum-redundancy codes , 1952, Proceedings of the IRE.

[16]  Jarek Duda,et al.  Asymmetric numeral systems , 2009, ArXiv.

[17]  Brian Parker Tunstall,et al.  Synthesis of noiseless compression codes , 1967 .

[18]  R. Rice,et al.  Adaptive Variable-Length Coding for Efficient Compression of Spacecraft Television Data , 1971 .