Modeling and simulation of distributed computing workflows in heterogeneous network environments

Next-generation computation- and network-intensive collaborative applications in various science, engineering, and e-commerce fields feature large-scale computing workflows of complex structures. Efficient algorithms are needed for task scheduling, module deployment, and service provisioning to support the execution of such distributed workflows in heterogeneous network environments and optimize their end-to-end performance for fast system response or smooth data flow. However, deploying large-scale distributed applications in real network environments is extremely challenging due to the inherent dynamics in the reliability, availability, accessibility, and capacity of massively distributed system resources, which are typically shared among a broad community of users over the Internet or dedicated connections. We propose a simulation system to study the execution dynamics of distributed computing workflows and evaluate the network performance of workflow scheduling or mapping algorithms before actual deployment and experimentation. The proposed simulation system visually illustrates the dynamic execution process of workflows in network environments by simulating module execution on computer nodes and data transfer over network links in a completely distributed and parallel manner. Furthermore, the simulation system takes background traffic and workload into consideration to achieve a high level of simulation accuracy for distributed applications deployed in shared production network environments. We implement the simulation system using multi-threaded programming and conduct extensive testings on various mapping schemes using a large number of simulated workflows and networks. The simulation-based performance measurements are quantitatively confirmed by both the experimental observations collected in real networks and the theoretical results obtained by rigorous performance analysis based on well-defined mathematical models.

[1]  Liang Chen,et al.  Sedna: A BPEL-Based Environment for Visual Scientific Workflow Modeling , 2007, Workflows for e-Science, Scientific Workflows for Grids.

[2]  John A. Miller,et al.  Using simulation to facilitate effective workflow adaptation , 2002, Proceedings 35th Annual Simulation Symposium. SS 2002.

[3]  G. Racherla,et al.  PARSIT A Parallel Algorithm Recon guration SImulation Tool , 2003 .

[4]  Rajkumar Buyya,et al.  GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing , 2002, Concurr. Comput. Pract. Exp..

[5]  Xun-Li Wang,et al.  Spallation Neutron Source , 2001 .

[6]  Chase Qishi Wu,et al.  Optimizing Distributed Computing Workflows in Heterogeneous Network Environments , 2010, ICDCN.

[7]  Gregor von Laszewski,et al.  Workflow Concepts of the Java CoG Kit , 2005, Journal of Grid Computing.

[8]  Yu. V. Kapitonova,et al.  Hardware simulation in distributed computing systems: Methods and tools , 1995 .

[9]  Gerald Estrin.  The SARA system for computer architecture design and modelling , 1977, SIML.

[10]  Péter Kacsuk,et al.  Workflow-level parameter study management in multi-grid environments by the P-GRADE portal , 2006 .

[11]  Carole A. Goble,et al.  Taverna: a tool for building and running workflows of services , 2006, Nucleic Acids Res..

[12]  Wayne J. Davis,et al.  A New Simulation Tool for the Modeling and Control of Distributed Systems , 2002, Simul..

[13]  Chase Qishi Wu,et al.  Maximizing Workflow Throughput for Streaming Applications in Distributed Environments , 2010, 2010 Proceedings of 19th International Conference on Computer Communications and Networks.

[14]  Amit P. Sheth,et al.  Simulation modeling within workflow technology , 1995, WSC '95.

[15]  Radu Prodan,et al.  Scheduling of scientific workflows in the ASKALON grid environment , 2005, SGMD.

[16]  Amit P. Sheth,et al.  Semantic E-Workflow Composition , 2003, Journal of Intelligent Information Systems.

[17]  Yves Robert,et al.  Mapping pipeline skeletons onto heterogeneous platforms , 2007, J. Parallel Distributed Comput..

[18]  Ishfaq Ahmad,et al.  Dynamic Critical-Path Scheduling: An Effective Technique for Allocating Task Graphs to Multiprocessors , 1996, IEEE Trans. Parallel Distributed Syst..

[19]  Chase Qishi Wu,et al.  Automation and management of scientific workflows in distributed network environments , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW).

[20]  Kun-Lung Wu,et al.  SODA: An Optimizing Scheduler for Large-Scale Stream-Based Distributed Computer Systems , 2008, Middleware.

[21]  Yi Gu,et al.  Brief announcement: Efficient pipeline configuration in distributed heterogeneous computing environments , 2008, PODC 2008.

[22]  Rajkumar Buyya,et al.  A Dynamic Critical Path Algorithm for Scheduling Scientific Workflow Applications on Global Grids , 2007, Third IEEE International Conference on e-Science and Grid Computing (e-Science 2007).

[24]  Gerti Kappel,et al.  TriGS/sub flow/: Active object-oriented workflow management , 1995, Proceedings of the Twenty-Eighth Annual Hawaii International Conference on System Sciences.

[25]  Ahmed Amine Jerraya,et al.  MCI- Multilanguage Distributed Co- Simulation Tool , 1998, DIPES.

[26]  Chase Qishi Wu,et al.  Optimizing End-to-end Performance of Distributed Applications with Linear Computing Pipelines , 2009, 2009 15th International Conference on Parallel and Distributed Systems.

[27]  E. Willen,et al.  Relativistic Heavy Ion Collider , 1986 .

[28]  Chase Qishi Wu,et al.  Simulation-based analysis of performance dynamics of distributed applications in heterogeneous network environments , 2009, SpringSim '09.

[29]  S. Rausch-Schott,et al.  TriGS flow Active Object-Oriented Workflow Management , 1995 .

[30]  Jason Maassen,et al.  Programming Scientific and Distributed Workflow with Triana Services , 2004 .

[31]  Edward A. Lee,et al.  Scientific workflow management and the Kepler system , 2006, Concurr. Comput. Pract. Exp..

[32]  Amit P. Sheth,et al.  An overview of workflow management: From process modeling to workflow automation infrastructure , 1995, Distributed and Parallel Databases.

[33]  Mitja Lenic,et al.  Web process and workflow path mining using the Multimethod approach , 2006, Int. J. Bus. Intell. Data Min..

[34]  Yechiam Yemini,et al.  NEST: a network simulation and prototyping testbed , 1990, CACM.

[35]  Umakishore Ramachandran,et al.  Streamline: a scheduling heuristic for streaming applications on the grid , 2006, Electronic Imaging.

[36]  Daniel S. Katz,et al.  Pegasus: A framework for mapping complex scientific workflows onto distributed systems , 2005, Sci. Program..

[37]  James C. Browne,et al.  General approach to mapping of parallel computations upon multiprocessor architectures , 1988 .

[38]  Jean-Luc Dekeyser,et al.  SOAP Based Distributed Simulation Environment for System-on-Chip (SoC) Design , 2005 .

[39]  Chase Qishi Wu,et al.  Latency modeling and minimization for large-scale scientific workflows in distributed network environments , 2011, SpringSim.

[40]  Chase Qishi Wu,et al.  Efficient pipeline configuration in distributed heterogeneous computing environments , 2008, PODC '08.

[41]  Chase Qishi Wu,et al.  Supporting Distributed Application Workflows in Heterogeneous Computing Environments , 2008, 2008 14th IEEE International Conference on Parallel and Distributed Systems.

[42]  Qishi Wu,et al.  On optimization of scientific workflows to support streaming applications in distributed network environments , 2010, The 5th Workshop on Workflows in Support of Large-Scale Science.

[43]  Li Zhao,et al.  Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance Tracking: The CyberShake Example , 2006, 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science'06).

[44]  Herwig Unger,et al.  SIMULATION OF COMMUNITIES OF NODES IN A WIDE AREA DISTRIBUTED SYSTEM , 2001 .