Scheduling big data applications within advance reservation framework in optical grids

Providing QoS for big data applications requires a way to reserve computing and networking resources in advance. Within advance reservation framework, a multi-domain scheduling process is carried out in a top down hierarchical way across multiple hierarchical levels. This ensures that each domain executes intra-domain scheduling algorithm to co-schedule its own computing and networking resources while coordinating the scheduling at the inter-domain level. Within this process, we introduce two algorithms: iterative scheduling algorithm and K-shortest paths algorithm. We conducted a comprehensive performance evaluation study considering several metrics that reflect both grid system and grid user goals. The results demonstrated the advantages of the proposed scheduling process. Moreover, the results highlight the importance of using the iterative scheduling and K-shortest paths algorithms especially for data intensive applications.

[1]  María Blanca Caminero,et al.  Using Network Information to Perform Meta-scheduling in Advance in Grids , 2010, Euro-Par.

[2]  Mohamed El-Darieby,et al.  Resource information sharing in multi-domain optical grids , 2014, Int. J. Grid Util. Comput..

[3]  Michiaki Hayashi,et al.  G-lambda: Coordination of a Grid Scheduler and Lambda Path Service over GMPLS , 2006, 2006 European Conference on Optical Communications.

[4]  Gigi Karmous-Edwards,et al.  Dynamic scheduling of network resources with advance reservations in optical grids , 2008, Int. J. Netw. Manag..

[5]  Mohamed El-Darieby,et al.  Co-Scheduling Computational and Networking Resources in E-Science Optical Grids , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[6]  Thomas Röblitz.  Global Optimization For Scheduling Multiple Co-Reservations In The Grid , 2008 .

[7]  Gabriel Junyent,et al.  Extending the UCLP software with a Dynamic Optical Multicast Service to support high performance digital media , 2008, 2008 International Conference on Optical Network Design and Modeling.

[8]  Yufeng Xin,et al.  EnLIGHTened Computing: An architecture for co-allocating network, compute, and other grid resources for high-end applications , 2007, 2007 International Symposium on High Capacity Optical Networks and Enabling Technologies.

[9]  Daniel S. Katz,et al.  Abstractions for Distributed Systems (DPA 2008) , 2008, Euro-Par Workshops.

[10]  Mohamed El-Darieby,et al.  Multidomain Hierarchical Resource Allocation for Grid Applications , 2012, J. Electr. Comput. Eng..

[11]  Tao Jiang,et al.  Application-specific resource provisioning for wide-area distributed computing , 2010, IEEE Network.

[12]  Peter Martini,et al.  Harmony - Advance Reservations in Heterogeneous Multi-domain Environments , 2009, Networking.

[13]  Mohamed El-Darieby,et al.  Towards Network-Aware Divisible Load Theory for Optical Grids , 2011, 2011 IEEE International Conference on High Performance Computing and Communications.

[14]  Mohamed El-Darieby,et al.  Load-Balancing Data Traffic Among Inter-Domain Links , 2007, IEEE Journal on Selected Areas in Communications.

[15]  María Blanca Caminero,et al.  Network-aware meta-scheduling in advance with autonomous self-tuning system , 2011, Future Gener. Comput. Syst..

[16]  Bruno Volckaert,et al.  Dimensioning and on-line scheduling in Lambda Grids using divisible load concepts , 2007, The Journal of Supercomputing.

[17]  Yanbin Liu,et al.  Looking for an Evolution of Grid Scheduling: Meta-Brokering , 2008 .

[18]  Henri Casanova,et al.  A realistic network/application model for scheduling divisible loads on large-scale platforms , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[19]  George N. Rouskas,et al.  Efficient resource management using advance reservations for heterogeneous Grids , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[20]  S. Figuerola,et al.  PHOSPHORUS: single-step on-demand services across multi-domain networks for e-science , 2007, SPIE/OSA/IEEE Asia Communications and Photonics.

[21]  Neal Charbonneau,et al.  Advance reservation frameworks in hybrid IP-WDM networks , 2011, IEEE Communications Magazine.

[22]  Admela Jukan,et al.  Evolution of Optical Networking Toward Rich Digital Media Services , 2012, Proceedings of the IEEE.

[23]  Johan Tordsson,et al.  A standards-based Grid resource brokering service supporting advance reservations, coallocation, and cross-Grid interoperability , 2009 .