An Elastic Middleware Platform for Concurrent and Distributed Cloud and MapReduce Simulations

Cloud Computing researches involve a tremendous amount of entities such as users, applications, and virtual machines. Due to the limited access and often variable availability of such resources, researchers have their prototypes tested against the simulation environments, opposed to the real cloud environments. Existing cloud simulation environments such as CloudSim and EmuSim are executed sequentially, where a more advanced cloud simulation tool could be created extending them, leveraging the latest technologies as well as the availability of multi-core computers and the clusters in the research laboratories. While computing has been evolving with multi-core programming, MapReduce paradigms, and middleware platforms, cloud and MapReduce simulations still fail to exploit these developments themselves. This research develops Cloud2Sim, which tries to fill the gap between the simulations and the actual technology that they are trying to simulate. First, Cloud2Sim provides a concurrent and distributed cloud simulator, by extending CloudSim cloud simulator, using Hazelcast in-memory key-value store. Then, it also provides a quick assessment to MapReduce implementations of Hazelcast and Infinispan, adaptively distributing the execution to a cluster, providing means of simulating MapReduce executions. The dynamic scaler solution scales out the cloud and MapReduce simulations to multiple nodes running Hazelcast and Infinispan, based on load. The distributed execution model and adaptive scaling solution could be leveraged as a general purpose auto scaler middleware for a multi-tenanted deployment.

[1]  Rajkumar Buyya,et al.  CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms , 2011, Softw. Pract. Exp..

[2]  Kavitha Ranganathan,et al.  Decoupling computation and data scheduling in distributed data-intensive applications , 2002, Proceedings 11th IEEE International Symposium on High Performance Distributed Computing.

[3]  Anita Weismantel Mikasa,et al.  Play it again , 1995 .

[4]  Bernhard Lwenstein.  Benchmarking of Middleware Systems: Evaluating and Comparing the Performance and Scalability of XVSM (MozartSpaces), JavaSpaces (GigaSpaces XAP) and J2EE (JBoss AS) , 2010 .

[5]  Gilles Fedak,et al.  Towards Making BOINC and EGEE Interoperable , 2008, 2008 IEEE Fourth International Conference on eScience.

[6]  Soumendra Mohanty,et al.  Extracting Value From Big Data: In-Memory Solutions, Real Time Analytics, And Recommendation Systems , 2013 .

[7]  Márk Jelasity,et al.  PeerSim: A scalable P2P simulator , 2009, 2009 IEEE Ninth International Conference on Peer-to-Peer Computing.

[8]  Rajesh Raman,et al.  Resource management through multilateral matchmaking , 2000, Proceedings the Ninth International Symposium on High-Performance Distributed Computing.

[9]  Jan Broeckhove,et al.  Improving the Scalability of SimGrid Using Dynamic Routing , 2009, ICCS.

[10]  A. Varga,et al.  THE OMNET++ DISCRETE EVENT SIMULATION SYSTEM , 2003 .

[11]  Yanpei Chen,et al.  Energy efficiency for large-scale MapReduce workloads with significant interactive analysis , 2012, EuroSys '12.

[12]  Sanjay Ghemawat,et al.  MapReduce: a flexible data processing tool , 2010, CACM.

[13]  Bhaskar Prasad Rimal,et al.  Grid, SOA and Cloud Computing: On-Demand Computing Models , 2012 .

[14]  Rajkumar Buyya,et al.  NetworkCloudSim: Modelling Parallel Applications in Cloud Simulations , 2011, 2011 Fourth IEEE International Conference on Utility and Cloud Computing.

[15]  Henri Casanova,et al.  SimGrid: A Generic Framework for Large-Scale Distributed Experiments , 2008, Tenth International Conference on Computer Modeling and Simulation (uksim 2008).

[16]  Virginia Mary Lo,et al.  2004 Ieee International Symposium on Cluster Computing and the Grid Cluster Computing on the Fly: Resource Discovery in a Cycle Sharing Peer-to-peer System , 2022 .

[17]  Roy H. Campbell,et al.  Play It Again, SimMR! , 2011, 2011 IEEE International Conference on Cluster Computing.

[18]  Dzmitry Kliazovich,et al.  GreenCloud: a packet-level simulator of energy-aware cloud computing data centers , 2010, The Journal of Supercomputing.

[19]  Luís E. T. Rodrigues,et al.  Exploiting Total Order Multicast in Weakly Consistent Transactional Caches , 2011, 2011 IEEE 17th Pacific Rim International Symposium on Dependable Computing.

[20]  Jacques Ferber,et al.  Multi-agent systems - an introduction to distributed artificial intelligence , 1999 .

[21]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[22]  Tharam S. Dillon,et al.  Cloud Computing: Issues and Challenges , 2010, 2010 24th IEEE International Conference on Advanced Information Networking and Applications.

[23]  Luciana Arantes,et al.  MRSG - A MapReduce simulator over SimGrid , 2013, Parallel Comput..

[24]  Guanying Wang,et al.  A simulation approach to evaluating design decisions in MapReduce setups , 2009, 2009 IEEE International Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication Systems.

[25]  Luís Veiga,et al.  Incremental replication for mobility support in OBIWAN , 2002, Proceedings 22nd International Conference on Distributed Computing Systems.

[26]  Richard M. Fujimoto.  Parallel and distributed discrete event simulation: algorithms and applications , 1993, WSC '93.

[27]  Mark Falco,et al.  Oracle Coherence 3.5 , 2010 .

[28]  D. Birchall,et al.  Computational Fluid Dynamics , 2020, Radial Flow Turbocompressors.

[29]  Kristina Chodorow,et al.  MongoDB: The Definitive Guide , 2010 .

[30]  András Varga,et al.  An overview of the OMNeT++ simulation environment , 2008, SimuTools.

[31]  Henri Casanova,et al.  The SIMGRID Project Simulation and Deployment of Distributed Applications , 2006, 2006 15th IEEE International Conference on High Performance Distributed Computing.

[32]  Voratas Kachitvichyanukul,et al.  Dynamic scheduling II: fast simulation model for grid scheduling using HyperSim , 2003, WSC '03.

[33]  Satish K. Tripathi,et al.  Parallel and distributed simulation of discrete event systems , 1994 .

[34]  Ian T. Foster,et al.  Globus Toolkit Version 4: Software for Service-Oriented Systems , 2005, Journal of Computer Science and Technology.

[35]  R. M. Fujimoto,et al.  Parallel discrete event simulation , 1989, WSC '89.

[36]  Martin Sarnovsky,et al.  Cloud-based clustering of text documents using the GHSOM algorithm on the GridGain platform , 2013, 2013 IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI).

[37]  David P. Anderson,et al.  BOINC: a system for public-resource computing and storage , 2004, Fifth IEEE/ACM International Workshop on Grid Computing.

[38]  Andrew A. Chien,et al.  The MicroGrid: a Scientific Tool for Modeling Computational Grids , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[39]  Jan M. Rabaey,et al.  Energy aware routing for low energy ad hoc sensor networks , 2002, 2002 IEEE Wireless Communications and Networking Conference Record. WCNC 2002 (Cat. No.02TH8609).

[40]  Kurt Stockinger,et al.  OptorSim-A Grid Simulator for Studying Dynamic Data Replication Strategies , 2003 .

[41]  Ian T. Foster,et al.  Condor-G: A Computation Management Agent for Multi-Institutional Grids , 2004, Cluster Computing.

[42]  Ian Wakeman,et al.  Towards Yet Another Peer-to-Peer Simulator , 2006 .

[43]  Henri Casanova,et al.  Simgrid: a toolkit for the simulation of application scheduling , 2001, Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid.

[44]  Pedro García López,et al.  PlanetSim: A New Overlay Network Simulation Framework , 2004, SEM.

[45]  Luís Veiga,et al.  nuBOINC: BOINC Extensions for Community Cycle Sharing , 2008, 2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops.

[46]  Luís Veiga,et al.  Transparent adaptation of e-science applications for parallel and cycle-sharing infrastructures , 2011, DAIS'11.

[47]  Rajkumar Buyya,et al.  Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in Cloud data centers , 2012, Concurr. Comput. Pract. Exp..

[48]  S. Krause,et al.  OverSim: A Flexible Overlay Network Simulation Framework , 2007, 2007 IEEE Global Internet Symposium.

[49]  Hai Jin,et al.  Carbon-Aware Load Balancing for Geo-distributed Cloud Services , 2013, 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems.

[50]  Luís Veiga,et al.  GiGi: An Ocean of Gridlets on a "Grid-for-the-Masses" , 2007, Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07).

[51]  Rajesh Raman,et al.  Matchmaking: distributed resource management for high throughput computing , 1998, Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244).

[52]  Henri Casanova,et al.  Scheduling distributed applications: the SimGrid simulation framework , 2003, CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings..

[53]  Charles L. Seitz,et al.  Variants of the Chandy-Misra-Bryant Distributed Discrete-Event Simulation Algorithm , 1988 .

[54]  Guanying Wang,et al.  Using realistic simulation for performance analysis of mapreduce setups , 2009, LSAP '09.

[55]  Roberto Palmieri,et al.  Integrated Monitoring of Infrastructures and Applications in Cloud Environments , 2011, Euro-Par Workshops.

[56]  Rajkumar Buyya,et al.  CloudAnalyst: A CloudSim-Based Visual Modeller for Analysing Cloud Computing Environments and Applications , 2010, 2010 24th IEEE International Conference on Advanced Information Networking and Applications.

[57]  Vijay S. Pande,et al.  Folding@home: Lessons from eight years of volunteer distributed computing , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[58]  Jayadev Misra,et al.  Distributed discrete-event simulation , 1986, CSUR.

[59]  Ian T. Foster,et al.  GangSim: a simulator for grid scheduling studies , 2005, CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005..

[60]  M. Samovsky,et al.  Cloud-based classification of text documents using the Gridgain platform , 2012, 2012 7th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI).

[61]  Richard Wolski,et al.  The Eucalyptus Open-Source Cloud-Computing System , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[62]  David Wolinsky,et al.  PonD: dynamic creation of HTC pool on demand using a decentralized resource discovery system , 2012, HPDC '12.

[63]  Vijay S. Pande,et al.  Folding@Home and Genome@Home: Using distributed computing to tackle previously intractable problem , 2009, 0901.0866.

[64]  Luís E. T. Rodrigues,et al.  Goal-oriented Self-management of In-memory Distributed Data Grid Platforms , 2011, 2011 IEEE Third International Conference on Cloud Computing Technology and Science.

[65]  David Wolinsky,et al.  Parallel Processing Framework on a P2P System Using Map and Reduce Primitives , 2011, 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum.

[66]  Mohamed F. Younis,et al.  Energy-aware routing in cluster-based sensor networks , 2002, Proceedings. 10th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunications Systems.

[67]  Miron Livny,et al.  Condor-a hunter of idle workstations , 1988, [1988] Proceedings. The 8th International Conference on Distributed.

[68]  Ewa Deelman,et al.  WorkflowSim: A toolkit for simulating scientific workflows in distributed environments , 2012, 2012 IEEE 8th International Conference on E-Science.

[69]  José Simão,et al.  Resource-Aware Scaling of Multi-threaded Java Applications in Multi-tenancy Scenarios , 2013, 2013 IEEE 5th International Conference on Cloud Computing Technology and Science.

[70]  Rajkumar Buyya,et al.  CloudSim: A Novel Framework for Modeling and Simulation of Cloud Computing Infrastructures and Services , 2009, ArXiv.

[71]  Rajkumar Buyya,et al.  Modeling and simulation of scalable Cloud computing environments and the CloudSim toolkit: Challenges and opportunities , 2009, 2009 International Conference on High Performance Computing & Simulation.

[72]  Chita R. Das,et al.  MDCSim: A multi-tier data center simulation, platform , 2009, 2009 IEEE International Conference on Cluster Computing and Workshops.

[73]  Vijay S. Pande,et al.  Screen Savers of the World Unite! , 2000, Science.

[74]  Arnaud Legrand,et al.  Accuracy study and improvement of network simulation in the SimGrid framework , 2009, SimuTools.

[75]  George Wells.  Coordination Languages: Back to the Future with Linda , 2005 .

[76]  David P. Anderson,et al.  SETI@home: an experiment in public-resource computing , 2002, CACM.

[77]  Ying He,et al.  ChordMR: A P2P-based Job Management Scheme in Cloud , 2014, J. Networks.

[78]  R.Martini,et al.  High-Availability Monitoring and Big Data: Using Java Clustering and Caching Technologies to Meet Complex Monitoring Scenarios , 2013 .

[79]  Rajiv Ranjan,et al.  G-Hadoop: MapReduce across distributed data centers for data-intensive computing , 2013, Future Gener. Comput. Syst..

[80]  Arnold H. Buss.  Modeling with event graphs , 1996, Winter Simulation Conference.

[81]  Christine Morin,et al.  Virtual Organization Support within a Grid-Wide Operating System , 2008, IEEE Internet Computing.

[82]  Erwin Laure,et al.  Enabling Grids for e-Science : The EGEE Project , 2008, Grid 2008.

[83]  Alec Wolman,et al.  Volley: Automated Data Placement for Geo-Distributed Cloud Services , 2010, NSDI.

[84]  Stefania Bandini,et al.  Dealing with space in multi--agent systems: a model for situated MAS , 2002, AAMAS '02.

[85]  Rajkumar Buyya,et al.  EMUSIM: an integrated emulation and simulation environment for modeling, evaluation, and validation of performance of Cloud computing applications , 2013, Softw. Pract. Exp..

[86]  Henri Casanova,et al.  Speed and accuracy of network simulation in the SimGrid framework , 2007, ValueTools '07.

[87]  Manas Ranjan Patra,et al.  An Adaptive Scheduling Mechanism for Computational Desktop Grid Using GridGain , 2012 .

[88]  Siddhartha Kumar Khaitan,et al.  High performance computing in power and energy systems , 2013 .

[89]  Peter M. A. Sloot,et al.  Introducing Grid Speedup G: A Scalability Metric for Parallel Applications on the Grid , 2005, EGC.

[90]  Ian T. Foster,et al.  The anatomy of the grid: enabling scalable virtual organizations , 2001, Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid.

[91]  Maozhen Li,et al.  HSim: A MapReduce simulator in enabling Cloud Computing , 2013, Future Gener. Comput. Syst..

[92]  Kazuyuki Shudo,et al.  Overlay Weaver: An overlay construction toolkit , 2008, Computer Communications.

[93]  Gunjan Ansari,et al.  Improved Performance of Advance Encryption Standard using Parallel Computing , 2012 .

[94]  Liang Liu,et al.  GreenCloud: a new architecture for green data center , 2009, ICAC-INDST '09.

[95]  Hanan Lutfiyya,et al.  DCSim: A data centre simulation tool for evaluating dynamic virtualized resource management , 2012, 2012 8th international conference on network and service management (cnsm) and 2012 workshop on systems virtualiztion management (svm).

[96]  Luís Veiga,et al.  An Adaptive Distributed Simulator for Cloud and MapReduce Algorithms and Architectures , 2014, 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing.

[97]  Henri Casanova,et al.  Fast and scalable simulation of volunteer computing systems using SimGrid , 2010, HPDC '10.

[98]  Peter T. L. Popkowski Leszczyc,et al.  The university of alberta. , 1969, Canadian family physician Medecin de famille canadien.

[99]  Luís Veiga,et al.  Concurrent and Distributed CloudSim Simulations , 2014, 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and Telecommunication Systems.

[100]  Mario Antonioletti,et al.  eScience, 2008. eScience '08. IEEE Fourth International Conference on , 2008 .

[101]  Ian Lumb,et al.  A Taxonomy and Survey of Cloud Computing Systems , 2009, 2009 Fifth International Joint Conference on INC, IMS and IDC.

[102]  Antonio Vallecillo,et al.  Concurrent Model Transformations with Linda , 2013 .

[103]  Daniel Mahrenholz,et al.  Real-Time Network Emulation with ns-2 , 2004, Eighth IEEE International Symposium on Distributed Simulation and Real-Time Applications.

[104]  Ekasit Kijsipongse,et al.  Scaling HPC Clusters with volunteer computing for data intensive applications , 2013, The 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE).

[105]  Jesús Carretero,et al.  iCanCloud: A Flexible and Scalable Cloud Infrastructure Simulator , 2012, Journal of Grid Computing.

[106]  Stefania Bandini,et al.  TOWARD A PLATFORM FOR MULTI-LAYERED MULTI-AGENT SITUATED SYSTEM (MMASS)-BASED SIMULATIONS: FOCUSING ON FIELD DIFFUSION , 2006, Appl. Artif. Intell..

[107]  Ayan Banerjee,et al.  GDCSim: A tool for analyzing Green Data Center design and resource management techniques , 2011, 2011 International Green Computing Conference and Workshops.

[108]  Rajkumar Buyya,et al.  Energy-aware resource allocation heuristics for efficient management of data centers for Cloud computing , 2012, Future Gener. Comput. Syst..