论文信息 - CloudWF: A Computational Workflow System for Clouds Based on Hadoop

CloudWF: A Computational Workflow System for Clouds Based on Hadoop

This paper describes CloudWF, a scalable and lightweight computational workflow system for clouds on top of Hadoop. CloudWF can run workflow jobs composed of multiple Hadoop MapReduce or legacy programs. Its novelty lies in several aspects: a simple workflow description language that encodes workflow blocks and block-to-block dependencies separately as standalone executable components; a new workflow storage method that uses Hadoop HBase sparse tables to store workflow information internally and reconstruct workflow block dependencies implicitly for efficient workflow execution; transparent file staging with Hadoop DFS; and decentralized workflow execution management relying on the MapReduce framework for task scheduling and fault tolerance. This paper describes the design and implementation of CloudWF.

Hans De Sterck | Chen Zhang | Chen Zhang | H. Sterck

[1] Rajkumar Buyya,et al. A taxonomy of scientific workflow systems for grid computing , 2005, SGMD.

[2] Hans De Sterck,et al. Case Study of Scientific Data Processing on a Cloud Using Hadoop , 2009, HPCS.

[3] Ewa Deelman,et al. The cost of doing science on the cloud: the Montage example , 2008, HiPC 2008.

[4] Rajkumar Buyya,et al. A Decentralized and Cooperative Workflow Scheduling Algorithm , 2008, 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid (CCGRID).

[5] Edward A. Lee,et al. CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE Concurrency Computat.: Pract. Exper. 2000; 00:1–7 Prepared using cpeauth.cls [Version: 2002/09/19 v2.02] Taverna: Lessons in creating , 2022 .

[6] Edward A. Lee,et al. Scientific workflow management and the Kepler system , 2006, Concurr. Comput. Pract. Exp..

[7] Lois M. L. Delcambre. Conceptual Modeling - ER 2005, 24th International Conference on Conceptual Modeling, Klagenfurt, Austria, October 24-28, 2005, Proceedings , 2005, ER.

[8] Bertram Ludäscher,et al. Actor-Oriented Design of Scientific Workflows , 2005, ER.

[9] Matthew S. Shields,et al. Triana: a graphical Web service composition and execution toolkit Web Services , 2004 .

[10] Anastasia Ailamaki,et al. Scientific workflow management by database management , 1998, Proceedings. Tenth International Conference on Scientific and Statistical Database Management (Cat. No.98TB100243).