A mobile code collaboration framework for grid computing

Data intensive scientific applications often involve diverse, high volume, and distributed data sets. They can generally be viewed as job workflows in which subjobs (i.e., nodes or activities) represent application components and dependencies represent the interactions between the components. To reduce the communication overhead caused by data movement and to provide decentralized control of execution during the workflow enactment, the Mobile Code Collaboration Framework (MCCF) is developed in this thesis to map the execution of subjobs to the distributed resources and to coordinate the subjobs’ execution at runtime according to the abstract workflow provided by users. Light-weight Mobile Agent (LMA) and Code-on-Demand (CoD) techniques are adopted in the development of the MCCF, so that an analysis module in data intensive scientific applications can be executed at a computational resource close to where the required data set is located. The MCCF, which does not have a centralized engine, is different from the existing scientific workflow engines (e.g., Condor’s DAGMan, SCIRun, Triana, and Taverna). When multiple data independent subjobs can be executed concurrently, replicas of an exiting LMA will be generated so that there is one LMA for each subjob. The LMAs will then be migrated to the different computational resources for the execution of these data independent subjobs in parallel. Because of the data dependencies in a job workflow, before an LMA executes a subjob, it needs to locate the execution results of its predecessors. When multiple concurrently executing subjobs have a common immediate successor, only one of the corresponding LMAs should be selected for the latter’s execution. Others should be discarded if they are not migrated to any successors’ execution. Due to the lack of a centralized engine, execution coordination is therefore required in the MCCF. In addition, the distributed nature of the MCCF also gives rise to the requirement of a distributed algorithm for various LMAs working collaboratively to collect complete, agi ATTENTION: The Singapore Copyright Act applies to the use of this document. Nanyang Technological University Library