Platform Overlays: enabling in-network stream processing in large-scale distributed applications

The purpose of this research is to explore the capabilities of future, multi-core heterogeneous systems, with specialized communication support, to be used as efficient and flexible execution platforms in distributed streaming applications. On such platforms, we create overlays of hardware- and software-supported execution contexts -- platform overlays. Stream manipulations, represented via stream handlers, are deployed on top of such overlays, based on the ability of individual contexts to perform handler operations. As a result, stream processing is dynamically mapped to those platform resources best suited for it, and it can even be fully contained to the networking subsystems, thereby enabling in-network stream processing. Experimental results demonstrate the benefits of our approach towards meeting application-specific quality requirements.

[1]  Michael Harville,et al.  An architecture for componentized, network-based media services , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[2]  Dhabaleswar K. Panda,et al.  High Performance RDMA-Based MPI Implementation over InfiniBand , 2003, ICS '03.

[3]  Bhaskaran Raman.  An architecture for highly available wide-area service composition , 2003, Comput. Commun..

[4]  Debanjan Saha,et al.  Design, implementation and performance of a content-based switch , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[5]  John W. Lockwood,et al.  Protocol Wrappers for Layered Network Packet Processing in Reconfigurable Hardware , 2002, IEEE Micro.

[6]  Yuanyuan Zhao,et al.  Exploitng event stream interpretation in publish-subscribe systems , 2001, PODC '01.

[7]  Jeffrey S. Chase,et al.  Payload Caching: High-Speed Data Forwarding for Network Intermediaries , 2001, USENIX ATC, General Track.

[8]  Dhabaleswar K. Panda,et al.  Can user-level protocols take advantage of multi-CPU NICs? , 2002, Proceedings 16th International Parallel and Distributed Processing Symposium.

[9]  Calton Pu,et al.  Operational information systems: an example from the airline industry , 2000, WIESS'00.

[10]  Ada Gavrilovska,et al.  Network processors as building blocks in overlay networks , 2003, 11th Symposium on High Performance Interconnects, 2003. Proceedings..

[11]  William Thies,et al.  StreamIt: A Language for Streaming Applications , 2002, CC.

[12]  Xiao Zhang,et al.  Anatomy of UDP and M-VIA for cluster communication , 2005, J. Parallel Distributed Comput..

[13]  Xuxian Jiang,et al.  Towards an integrated multimedia service hosting overlay , 2004, MULTIMEDIA '04.

[14]  Eddie Kohler,et al.  The Click modular router , 1999, SOSP.

[15]  Yitzchak M. Gottlieb,et al.  Building a robust software-based router using network processors , 2001, SOSP.

[16]  Yuanyuan Zhao yuanyuan.  Exploiting Event Stream Interpretation in Publish-Subscribe Systems , 2001 .

[17]  Karsten Schwan,et al.  KStreams: kernel support for efficient data streaming in proxy servers , 2005, NOSSDAV '05.

[18]  Calton Pu,et al.  Spidle: A DSL Approach to Specifying Streaming Applications , 2003, GPCE.

[19]  Vikram A. Saletore,et al.  ETA: experience with an Intel Xeon processor as a packet processing engine , 2004, IEEE Micro.