Adaptive online data compression

Quickly transmitting large datasets in the context of distributed computing on wide area networks can be achieved by compressing data before transmission, However such an approach is not efficient when dealing with higher speed networks. Indeed, the time to compress a large file and to send it is greater than the time to send the uncompressed file. In this paper we explore and enhance an algorithm that allows us to overlap communications with compression and to automatically adapt the compression effort to currently available network and processor resources.

[1]  Daniel S. Hirschberg,et al.  Data compression , 1987, CSUR.

[2]  Pierre Ramet,et al.  Optimal Grain Size Computation for Pipelined Algorithms , 1996, Euro-Par, Vol. I.

[3]  John Woods.  PPP Deflate Protocol , 1996, RFC.

[4]  Aladdin Enterprises,et al.  ZLIB Compressed Data Format Specification version 3.3 , 1996 .

[5]  Björn Knutsson,et al.  Architectures for application transparent proxies : A study of network enhancing software , 2001 .

[6]  Maurice Goursat,et al.  SCILAB to SCILAB//: The OURAGAN project , 2001, Parallel Comput..

[7]  Peter Deutsch,et al.  GZIP file format specification version 4.3 , 1996, RFC.

[8]  Mats Björkman,et al.  Adaptive end-to-end compression for variable-bandwidth communication , 1999, Comput. Networks.

[9]  Vernon Schryver.  PPP BSD Compression Protocol , 1996, RFC.

[10]  Gurdeep Singh Pall.  Microsoft Point-To-Point Compression (MPPC) Protocol , 1997, RFC.

[11]  Dave Rand.  The PPP Compression Control Protocol (CCP) , 1996, RFC.

[12]  Mitsuhisa Sato,et al.  Ninf: A Network Based Information Library for Global World-Wide Computing Infrastructure , 1997, HPCN Europe.

[13]  Henri Casanova,et al.  NetSovle: A Network Server for Solving Computational Science Problems , 1996, Proceedings of the 1996 ACM/IEEE Conference on Supercomputing.

[14]  Abraham Lempel,et al.  A universal algorithm for sequential data compression , 1977, IEEE Trans. Inf. Theory.

[15]  Abraham Lempel,et al.  Compression of individual sequences via variable-rate coding , 1978, IEEE Trans. Inf. Theory.

[16]  Terry A. Welch,et al.  A Technique for High-Performance Data Compression , 1984, Computer.

[17]  Jonathan M. Smith,et al.  Operating System Support for Protocol Boosters , 1996 .

[18]  Fred Douglis,et al.  The Compression Cache: Using On-line Compression to Extend Physical Memory , 1993, USENIX Winter.

[19]  David A. Huffman,et al.  A method for the construction of minimum-redundancy codes , 1952, Proceedings of the IRE.