Modelling and programming stream-based distributed computing based on the meta-pipeline approach

Caravela is a computing platform that has been designed in academia to implement distributed stream-based computing based on the flow-model. This paper is focused on the new meta-pipeline execution mechanism, which allows to set up a pipeline of flow-models to be executed in parallel but with data streams flowing in sequence along them. A model and programming tools are presented to develop the proposed meta-pipeline in the Caravela platform. These tools are important, namely to tackle the deadlock problem that can occur due to uninitialised input data streams. A new efficient algorithm is also proposed to identify off-line the uninitialised edges in a ‘pipeline graph’. Moreover, optimisation techniques are also proposed to reduce the computation time and the amount of memory required to implement the proposed algorithm, and a graphical user interface (GUI) was implemented to facilitate the programming of applications with the meta-pipeline approach. The experimental results presented in this paper show the practical interest of the meta-pipeline approach for stream-based computing and the effectiveness of both the proposed algorithm and the developed GUI for programming the Caravela platform.

[1]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[2]  R. V. van Nieuwpoort,et al.  The Grid 2: Blueprint for a New Computing Infrastructure , 2003 .

[3]  Pat Hanrahan,et al.  Brook for GPUs: stream computing on graphics hardware , 2004, ACM Trans. Graph..

[4]  Jens H. Krüger,et al.  A Survey of General‐Purpose Computation on Graphics Hardware , 2007, Eurographics.

[5]  Mendel Rosenblum,et al.  Stream programming on general-purpose processors , 2005, 38th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO'05).

[6]  Randi J. Rost.  OpenGL shading language , 2004 .

[7]  Leonel Sousa,et al.  Design and implementation of a tool for modeling and programming deadlock free meta-pipeline applications , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[8]  Leonel Sousa,et al.  Design and implementation of a stream-based distributedcomputing platform using graphics processing units , 2007, CF '07.

[9]  Leonel Sousa,et al.  Design and implementation of a graphical user interface for stream-based distributed computing , 2008 .

[10]  Jim X. Chen,et al.  OpenGL Shading Language , 2009 .

[11]  Leonel Sousa,et al.  Caravela: A Novel Stream-Based Distributed Computing Environment , 2007, Computer.

[12]  Leonel Sousa,et al.  Data buffering optimization methods toward a uniform programming interface for gpu-based applications , 2007, CF '07.

[13]  Leonel Sousa,et al.  Meta-Pipeline: A New Execution Mechanism for Distributed Pipeline Processing , 2007, Sixth International Symposium on Parallel and Distributed Computing (ISPDC'07).

[14]  Robert E. Tarjan.  Testing flow graph reducibility , 1973, STOC '73.

[15]  Ian Foster,et al.  The Grid 2 - Blueprint for a New Computing Infrastructure, Second Edition , 1998, The Grid 2, 2nd Edition.

[16]  Arie E. Kaufman,et al.  GPU Cluster for High Performance Computing , 2004, Proceedings of the ACM/IEEE SC2004 Conference.

[17]  Paul Havlak,et al.  Nesting of reducible and irreducible loops , 1997, TOPL.