Challenges in using scientific workflow tools in the hydrology domain

Scientific workflow tools are used to perform complex analysis on scientific data. The strength of scientific workflow tools lies in their ability to capture a complex analysis as a sequence of steps using simple components. The components may run on different computers or clusters located at different geographical locations and access data from heterogeneous sources. Scientific workflow tools are popular in specific scientific domains (e.g., ecology, genomics, and astrophysics) for the integration of simulation models. However, this kind of software framework has not been widely embraced by the hydrology domain. Typically hydrologists use a combination of off-the-shelf software systems such as The Invisible Modelling Environment (TIME) or Matlab along with other ad-hoc software to perform integration and to implement the workflow as their environment for hydrology simulations. In a large scale hydrology study, such as the Murray Darling Sustainable Yields undertaken by CSIRO, a substantial fraction of the overall cost is devoted to developing a collection of software tools to implement the processing flow required for the project. Scientific workflow tools offer significant potential to allow this software to be modularised, reused and shared. The process composition can be semi-automated. For an organisation such as the Bureau of Meteorology Water Division, which is responsible for the routine production of data products like a national water account, these tools can provide greatly improved transparency into the method and auditibility of the result. There are several challenges to the application of scientific workflow in hydrology domain. This paper explores some of these challenges: managing and processing large volumes of data, integration of heterogeneous data and model integration. Grid computing could be used as an interoperability platform to manage compute and data resources. Considerable work has been carried out in the grid computing community to develop methods to discover and efficiently access data by enabling services such as Fast Data Transfer, GridFTP and RapidFTP. We recommend the compute and data resources to be stored in a same grid infrastructure to provide effective execution. We also describe some applications that use grid computing to improve the execution speed of the hydrology models. Finally, we report on the adaptation of Kepler for hydrology domain. Kepler is a service-based workflow tool used extensively in some scientific domain (e.g. Ecology). We report on some preliminary work through modifications of existing actors to suit our needs and development of new actors that allow access to the Open Geospatial Consortium- Sensor Web Enablement web services.