A user-centric execution environment for CineGrid workloads

The abundance and heterogeneity of IT resources available, together with the ability to dynamically scale applications poses significant usability issues to users. Without understanding the performance profile of available resources users are unable to efficiently scale their applications in order to meet performance objectives. High quality media collaborations, like CineGrid, are one example of such diverse environments where users can leverage dynamic infrastructures to move and process large amounts of data. This paper describes our user-centric approach to executing high quality media processing workloads over dynamic infrastructures. Our main contribution is the CGtoolkit ?environment, an integrated system which aids users cope with the infrastructure complexity and large data sets specific to the digital cinema domain. We describe the general challenges user face in heterogeneous it infrastructure environments and identify their instance in the case of CineGrid.We characterize the most frequent workloads in the CineGrid collaboration including their infrastructure resource requirements.We present the design and implementation of a system which simplifies workload resource definition by leveraging semantic web description languages for infrastructure.We provide Pareto-optimal resource configurations for workload execution.

[1]  Cees T. A. M. de Laat,et al.  Towards an Infrastructure Description Language for Modeling Computing Infrastructures , 2012, 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications.

[2]  José A. B. Fortes,et al.  On the Use of Machine Learning to Predict the Time and Resources Consumed by Applications , 2010, 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing.

[3]  Henri Casanova,et al.  Non-clairvoyant Scheduling of Multiple Bag-of-Tasks Applications , 2010, Euro-Par.

[4]  A. Messac,et al.  The normalized normal constraint method for generating the Pareto frontier , 2003 .

[5]  Cees T. A. M. de Laat,et al.  A Queueing Theory Approach to Pareto Optimal Bags-of-Tasks Scheduling on Clouds , 2014, Euro-Par.

[6]  Dimitra Simeonidou,et al.  Cloud-based architecture for deploying ultra-high-definition media over intelligent optical networks , 2012, 2012 16th International Conference on Optical Network Design and Modelling (ONDM).

[7]  Tracy Cornish,et al.  Vroom: designing an augmented environment for remote collaboration in digital cinema production , 2013, Electronic Imaging.

[8]  Andreas Beyer,et al.  Dynamic resource allocation for cloud-based media processing , 2013, NOSSDAV '13.

[9]  Yves Robert,et al.  Scheduling Concurrent Bag-of-Tasks Applications on Heterogeneous Platforms , 2010, IEEE Transactions on Computers.

[10]  Guido Lemos de Souza Filho,et al.  A Software-Based Solution for Distributing and Displaying 3D UHD Films , 2013, IEEE MultiMedia.

[11]  Stephen S. Lavenberg,et al.  Computer Performance Modeling Handbook , 1983, Int. CMG Conference.

[12]  Cees T. A. M. de Laat,et al.  Using ontologies for resource description in the CineGrid Exchange , 2011, Future Gener. Comput. Syst..

[13]  Petr Holub,et al.  High-definition multimedia for multiparty low-latency interactive communication , 2006, Future Gener. Comput. Syst..

[14]  Alexandru Iosup,et al.  Trace-based evaluation of job runtime and queue wait time predictions in grids , 2009, HPDC '09.

[15]  Thilo Kielmann,et al.  Budget Estimation and Control for Bag-of-Tasks Scheduling in Clouds , 2011, Parallel Process. Lett..

[16]  Albert Y. Zomaya,et al.  Pareto-Optimal Cloud Bursting , 2014, IEEE Transactions on Parallel and Distributed Systems.

[17]  Alexandru Iosup,et al.  The performance of bags-of-tasks in large-scale distributed systems , 2008, HPDC '08.

[18]  Yves Robert,et al.  Steady-state scheduling on heterogeneous clusters , 2005, Int. J. Found. Comput. Sci..

[19]  Shaofeng Liu,et al.  CineGrid Exchange: A workflow-based peta-scale distributed storage platform on a high-speed network , 2011, Future Gener. Comput. Syst..

[20]  Alexandru Iosup,et al.  ExPERT: Pareto-Efficient Task Replication on Grids and a Cloud , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium.