Improving middleware performance with AdOC: an adaptive online compression library for data transfer

In this article, we present the AdOC (adaptive online compression) library. It is a user-level set of functions that enables data transmission with compression. The compression is performed dynamically during the transmission and the compression level is constantly adapted according to the environment. In order to ease the integration of AdOC into existing software the API is very close to the read and write UNIX system calls and respects their semantic. Moreover this library is thread-safe and is ported to many UNIX-like systems. We have tested AdOC under various conditions and with various data types. Results show that the library outperforms the POSIX read/write system calls on a broad range of networks (up to 100 Mbit LAN), whereas on Gbit Ethernet, it provides similar performance.

[1]  Henri Casanova,et al.  NetSovle: A Network Server for Solving Computational Science Problems , 1996, Proceedings of the 1996 ACM/IEEE Conference on Supercomputing.

[2]  Peter Deutsch,et al.  GZIP file format specification version 4.3 , 1996, RFC.

[3]  Abraham Lempel,et al.  Compression of individual sequences via variable-rate coding , 1978, IEEE Trans. Inf. Theory.

[4]  Tom Goodale,et al.  The Cactus computational collaboratory: enabling technologies for relativistic astrophysics, and a toolkit for solving PDE's by communities in science and engineering , 1999, Proceedings. Frontiers '99. Seventh Symposium on the Frontiers of Massively Parallel Computation.

[5]  W. Allcock,et al.  GridFTP protocol specification , 2002 .

[6]  Emmanuel Jeannot,et al.  Adaptive online data compression , 2002, Proceedings 11th IEEE International Symposium on High Performance Distributed Computing.

[7]  Dave Rand.  The PPP Compression Control Protocol (CCP) , 1996, RFC.

[8]  Mitsuhisa Sato,et al.  Ninf: A Network Based Information Library for Global World-Wide Computing Infrastructure , 1997, HPCN Europe.

[9]  J. Postel,et al.  File transfer protocol (FTP) , 1985 .

[10]  Alessandro Bassi,et al.  The Internet Backplane Protocol: A Study in Resource Sharing , 2002, 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGRID'02).

[11]  Kees Verstoep,et al.  Wide-area communication for grids: an integrated solution to connectivity, performance and security problems , 2004, Proceedings. 13th IEEE International Symposium on High performance Distributed Computing, 2004..

[12]  Karsten Schwan,et al.  Efficient end to end data exchange using configurable compression , 2004, 24th International Conference on Distributed Computing Systems, 2004. Proceedings..

[13]  G. Allen,et al.  Supporting Efficient Execution in Heterogeneous Distributed Computing Environments with Cactus and Globus , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[14]  Karsten Schwan,et al.  Efficient End to End Data Exchange Using Configurable Compression , 2004, ICDCS.

[15]  Frédéric Suter,et al.  A Scalable Approach to Network Enabled Servers , 2002, ASIAN.

[16]  Ian T. Foster,et al.  The Globus project: a status report , 1998, Proceedings Seventh Heterogeneous Computing Workshop (HCW'98).

[17]  Abraham Lempel,et al.  A universal algorithm for sequential data compression , 1977, IEEE Trans. Inf. Theory.

[18]  Jean-Marc Nicod,et al.  A Scalable Approach to Network Enabled Servers (Research Note) , 2002, Euro-Par.