Towards a provenance framework for sub-image processing for astronomical data

While there has been advances in observational equipment that generate huge high quality images, the processing of these images remains a major bottleneck. We show that provenance data collected during the processing of data can be reused to perform selective processing of data and support network collaboration without clogging distribution networks. We introduce the idea of sub-image processing (SIMP) in the context of processing a subset of pixels of an image and the use of provenance data to assemble pipelines and to select processing metadata for SIMP. We describe an implementation of SIMP in Astro-WISE

[1]  Perry Greenfield Reaching for the Stars with Python , 2007, Computing in Science & Engineering.

[2]  Edwin Valentijn,et al.  Astro-WISE: Tracing and Using Lineage for Scientific Data Processing , 2009, 2009 International Conference on Network-Based Information Systems.

[3]  Rajkumar Buyya,et al.  A taxonomy of scientific workflow systems for grid computing , 2005, SGMD.

[4]  Andrey N. Belikov,et al.  Merging Grid Technologies , 2010, Journal of Grid Computing.

[5]  Alexander S. Szalay,et al.  The Sloan Digital Sky Survey and beyond , 2008, SGMD.

[6]  Geoffrey C. Fox,et al.  Examining the Challenges of Scientific Workflows , 2007, Computer.

[7]  Scott Klasky,et al.  Introduction to scientific workflow management and the Kepler system , 2006, SC.

[8]  Simon Miles Automatically Adapting Source Code to Document Provenance , 2010, IPAW.

[9]  Cláudio T. Silva,et al.  Provenance for Computational Tasks: A Survey , 2008, Computing in Science & Engineering.

[10]  Cláudio T. Silva,et al.  Querying and Creating Visualizations by Analogy , 2007, IEEE Transactions on Visualization and Computer Graphics.