An Automatic and Scalable Testing Tool for Workflow Systems

Nowadays workflow systems are widely deployed around the world, especially within large international corporations. Thus the performance evaluation of these workflow systems becomes an important topic, especially in grid environment. We have developed an automatic and scalable testing tool to evaluate workflow systems. This tool is named Scalable Workflow systEm Evaluation Tool (SWEET). SWEET can automatically test against workflow systems according to a highly customizable process template specified by users. It can also provide multi-layer performance metrics related to response time, throughput and fairness. Further, it can adoptively determine the maximum concurrency of a workflow system under a desired average processing time, using control theory (PI Controller). In this paper, we will present the design and architecture of SWEET. We will also discuss the evaluation metrics and our implementation of such a framework. Finally, we will give detailed experimental results and analysis.

[1]  John A. Miller,et al.  Using simulation to facilitate effective workflow adaptation , 2002, Proceedings 35th Annual Simulation Symposium. SS 2002.

[2]  Mathias Weske,et al.  Advanced Topics in Workflow Management: Issues, Requirements, and solutions , 2003, Trans. SDPS.

[3]  Barbara S. Chaparro,et al.  The world wide wait: effects of delays on user performance , 2000 .

[4]  Armin Zimmermann,et al.  Modelling and evaluation of time aspects in business processes , 2002, J. Oper. Res. Soc..

[5]  J. Leon Zhao,et al.  Workflow Automation: Overview and Research Issues , 2001, Inf. Syst. Frontiers.

[6]  Amit P. Sheth,et al.  Workflow Quality of Service , 2002, ICEIMT.

[7]  Allan Kuchinsky,et al.  Integrating user-perceived quality into Web server design , 2000, Comput. Networks.

[8]  Schahram Dustdar,et al.  Performance metrics and ontology for describing performance data of grid workflows , 2005, CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005..

[9]  Michael Schroeder,et al.  An in-service agent monitoring and analysis system , 1999, Proceedings 11th International Conference on Tools with Artificial Intelligence.

[10]  Andrea F. Abate,et al.  Workflow performance evaluation through WPQL , 2002, SEKE '02.

[11]  Amit P. Sheth,et al.  Modeling Quality of Service for Workflows and Web Service Processes , 2002 .

[12]  Jianmin Zhao,et al.  A methodology for dynamic enterprise process performance evaluation , 2007, Comput. Ind..

[13]  Amit P. Sheth,et al.  Simulation modeling within workflow technology , 1995, WSC '95.