A communication-aware framework for parallel spatially explicit agent-based models

Parallel spatially explicit agent-based models (SE-ABM) exploit high-performance and parallel computing to simulate spatial dynamics of complex geographic systems. The integration of parallel SE-ABM with CyberGIS could facilitate straightforward access to massive computational resources and geographic information systems to support pre- and post-simulation analysis and visualization. However, to benefit from CyberGIS integration, parallel SE-ABM must overcome the challenge of communication management for orchestrating many processor cores in parallel computing environments. This paper examines and addresses this challenge by describing a generic framework for the management of inter-processor communication to enable parallel SE-ABM to scale to high-performance parallel computers. The framework synthesizes four interrelated components: agent grouping, rectilinear domain decomposition, a communication-aware load-balancing strategy, and entity proxies. The results of a series of computational experiments based on a template agent-based model demonstrate that parallel computational efficiency diminishes as inter-processor communication increases, particularly when scaling a fixed-size model to thousands of processor cores. Therefore, effective communication management is crucial. The communication framework is shown to efficiently scale up to 2048 cores, demonstrating its ability to effectively scale to thousands of processor cores to support the simulation of billions of agents. In a simulated scenario, the communication-aware load-balancer reduced both overall simulation time and communication percentage improving overall computational efficiency. By examining and addressing inter-processor communication challenges, this research enables parallel SE-ABM to efficiently use high-performance computing resources, which reduces the barriers for synergistic integration with CyberGIS.

[1]  Joshua M. Epstein,et al.  Modelling to contain pandemics , 2009, Nature.

[2]  Bernard P. Zeigler,et al.  Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems , 2000 .

[3]  Werner Dubitzky,et al.  An Overview of the SimWorld Agent-Based Grid Experimentation System , 2012 .

[4]  Veljko M. Milutinovic,et al.  Distributed shared memory: concepts and systems , 1997, IEEE Parallel Distributed Technol. Syst. Appl..

[5]  Ferdi L. Hellweger,et al.  Spatially explicit individual-based modeling using a fixed super-individual density , 2008, Comput. Geosci..

[6]  C. Linard,et al.  A multi-agent simulation to assess the risk of malaria re-emergence in southern France , 2008 .

[7]  Michael W. Berry,et al.  A Parallel Fish Landscape Model for Ecosystem Modeling , 2006, Simul..

[8]  Vijay P. Kumar,et al.  Analyzing Scalability of Parallel Algorithms and Architectures , 1994, J. Parallel Distributed Comput..

[9]  Hazel R. Parry Agent Based Modeling, Large Scale Simulations , 2009, Encyclopedia of Complexity and Systems Science.

[10]  Joshua M. Epstein,et al.  Growing Artificial Societies: Social Science from the Bottom Up , 1996 .

[11]  Georgios K. Theodoropoulos,et al.  Distributing RePast agent‐based simulations with HLA , 2008, Concurr. Comput. Pract. Exp..

[12]  Robert A. Meyers,et al.  Encyclopedia of Complexity and Systems Science , 2009 .

[13]  Bengt Aspvall,et al.  Approximations for the general block distribution of a matrix , 1998, Theor. Comput. Sci..

[14]  Michael W. Berry,et al.  On Parallelization of a Spatially-Explicit Structured Ecological Model for Integrated Ecosystem Simulation , 2006, Int. J. High Perform. Comput. Appl..

[15]  Kai Nagel,et al.  Parallel implementation of the TRANSIMS micro-simulation , 2001, Parallel Comput..

[16]  Kathleen M. Carley,et al.  BioWar: scalable agent-based model of bioattacks , 2006, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[17]  J. Neyman,et al.  Statistical Approach to Problems of Cosmology , 1958 .

[18]  Giandomenico Spezzano,et al.  Simulation of a cellular landslide model with CAMELOT on high performance computers , 2003, Parallel Comput..

[19]  Jon Parker A flexible, large-scale, distributed agent based epidemic model , 2007, 2007 Winter Simulation Conference.

[20]  Yuemin Ding,et al.  Spatial Strategies for Parallel Spatial Modelling , 1996, Int. J. Geogr. Inf. Sci..

[21]  M. Janssen,et al.  Multi-Agent Systems for the Simulation of Land-Use and Land-Cover Change: A Review , 2003 .

[22]  John L. Gustafson,et al.  Reevaluating Amdahl's law , 1988, CACM.

[23]  Christophe Le Page,et al.  A multi-agents architecture to enhance end-user individual-based modelling , 2002 .

[24]  Jeffrey A. Nichols,et al.  Parallel simulation of ecological structured communities : Computational needs, hardware capabilities, and nonlinear applications , 2008 .

[25]  V. Grimm Ten years of individual-based modelling in ecology: what have we learned and what could we learn in the future? , 1999 .

[26]  John L. Gustafson,et al.  Fixed Time, Tiered Memory, and Superlinear Speedup , 1990, Proceedings of the Fifth Distributed Memory Computing Conference, 1990..

[27]  Michael W. Berry,et al.  A Parallel Structured Ecological Model for High End Shared Memory Computers , 2005, IWOMP.

[28]  Sudip K. Seal,et al.  Discrete event modeling and massively parallel execution of epidemic outbreak phenomena , 2012, Simul..

[29]  Andrew Crooks,et al.  The Integration of Agent-Based Modelling and Geographical Information for Geospatial Simulation , 2012 .

[30]  Michelangelo Grigni,et al.  On the Complexity of the Generalized Block Distribution , 1996, IRREGULAR.

[31]  David O'Sullivan,et al.  Toward micro-scale spatial modeling of gentrification , 2002, J. Geogr. Syst..

[32]  William Rand,et al.  Spatial process and data models: Toward integration of agent-based models and GIS , 2005, J. Geogr. Syst..

[33]  Bin Li,et al.  A GIS based spatially explicit model of dispersal agent behavior , 2008 .

[34]  Graham Coates,et al.  Agent-based simulation for large-scale emergency response: A survey of usage and implementation , 2012, CSUR.

[35]  Shaowen Wang A CyberGIS Framework for the Synthesis of Cyberinfrastructure, GIS, and Spatial Analysis , 2010 .

[36]  David M. Nicol,et al.  Rectilinear Partitioning of Irregular Data Parallel Computations , 1994, J. Parallel Distributed Comput..

[37]  Steven L. Lytinen,et al.  Agent-based Simulation Platforms: Review and Development Recommendations , 2006, Simul..

[38]  David Skinner Performance monitoring of parallel scientific applications , 2005 .

[39]  Michael J. Quinn,et al.  PARALLEL IMPLEMENTATION OF THE SOCIAL FORCES MODEL , 2003 .

[40]  Jeffrey Shaman,et al.  Amplification due to spatial clustering in an individual-based model of mosquito-avian arbovirus transmission. , 2007, Transactions of the Royal Society of Tropical Medicine and Hygiene.

[41]  Uta Berger,et al.  Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology , 2005, Science.

[42]  Robert L. Goldstone,et al.  Computational models of collective behavior , 2005, Trends in Cognitive Sciences.

[43]  Kiril Stanilov,et al.  Space in Agent-Based Models , 2012 .

[44]  Shaowen Wang,et al.  HPABM: A Hierarchical Parallel Simulation Framework for Spatially‐explicit Agent‐based Models , 2009, Trans. GIS.

[45]  Alois Ferscha,et al.  Parallel and Distributed Simulation , 1996, Proceedings of HICSS-29: 29th Hawaii International Conference on System Sciences.

[46]  David A. Bennett,et al.  Toward an understanding of provenance in complex land use dynamics , 2011 .

[47]  Michael W. Berry,et al.  Toward ecosystem modeling on computing grids , 2005, Computing in Science & Engineering.

[48]  Laxmikant V. Kale,et al.  NAMD2: Greater Scalability for Parallel Molecular Dynamics , 1999 .

[49]  Werner Dubitzky,et al.  Large-Scale Computing Techniques for Complex System Simulations , 2011 .

[50]  Laxmikant V. Kalé,et al.  Understanding Application Performance via Micro-benchmarks on Three Large Supercomputers: Intrepid, Ranger and Jaguar , 2010, Int. J. High Perform. Comput. Appl..

[51]  P. Torrens Moving Agent Pedestrians Through Space and Time , 2012 .

[52]  Raymond R. Hill,et al.  A Survey of Agent-Based Modeling Practices (January 1998 to July 2008) , 2009, J. Artif. Soc. Soc. Simul..

[53]  Ling Bian,et al.  The representation of the environment in the context of individual-based modeling , 2003 .

[54]  Donald S. Fussell,et al.  Computer rendering of stochastic models , 1982, Commun. ACM.

[55]  Shaowen Wang,et al.  Agent-based modeling within a cyberinfrastructure environment: a service-oriented computing approach , 2011, Int. J. Geogr. Inf. Sci..

[56]  M. Haklay,et al.  Agent-Based Models and Individualism: Is the World Agent-Based? , 2000 .

[57]  Madhav V. Marathe,et al.  EpiSimdemics: An efficient algorithm for simulating the spread of infectious disease over large realistic social networks , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[58]  Hazel R. Parry,et al.  Large Scale Agent-Based Modelling: A Review and Guidelines for Model Scaling , 2012 .

[59]  Roshan M. D'Souza,et al.  A Framework for Megascale Agent Based Model Simulations on Graphics Processing Units , 2008, J. Artif. Soc. Soc. Simul..

[60]  Werner Dubitzky,et al.  Repast HPC: A Platform for Large-Scale Agent-Based Modeling , 2012 .

[61]  Wenwu Tang,et al.  Simulated village locations in Thailand: a multi-scale model including a neural network approach , 2009, Landscape Ecology.

[62]  Joshua M. Epstein,et al.  Growing Artificial Societies: Social Science from the Bottom Up , 1996 .

[63]  Jon Parker,et al.  A Distributed Platform for Global-Scale Agent-Based Models of Disease Transmission , 2011, TOMC.

[64]  Matthias Scheutz,et al.  Adaptive algorithms for the dynamic distribution and parallel execution of agent-based models , 2006, J. Parallel Distributed Comput..

[65]  Robert J. Fowler,et al.  NUMA policies and their relation to memory architecture , 1991, ASPLOS IV.

[66]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[67]  Michael W. Berry,et al.  Parallel individual-based modeling of Everglades deer ecology , 1997 .

[68]  Luc T. Wille,et al.  Parallelization of an ecological landscape model by functional decomposition , 2001 .

[69]  D. J. Tildesley,et al.  Large Scale Molecular Dynamics on Parallel Computers using the Link-cell Algorithm , 1991 .

[70]  J. A. Nichols,et al.  Parallel simulation of individual-based, physiologically structured population models , 1997 .

[71]  Shaowen Wang,et al.  A theoretical approach to the use of cyberinfrastructure in geographical analysis , 2009, Int. J. Geogr. Inf. Sci..

[72]  Michael Batty,et al.  Ucl Centre for Advanced Spatial Analysis Working Papers Series Key Challenges in Agent-based Modelling for Geo-spatial Simulation Paper 121 -sept 07 Key Challenges in Agent-based Modelling for Geo-spatial Simulation , 2022 .

[73]  Gregory R. Madey,et al.  Tools of the Trade: A Survey of Various Agent Based Modeling Platforms , 2009, J. Artif. Soc. Soc. Simul..

[74]  Ümit V. Çatalyürek,et al.  A repartitioning hypergraph model for dynamic load balancing , 2009, J. Parallel Distributed Comput..

[75]  Andrew Crooks,et al.  Agent-based Models of Geographical Systems , 2012 .

[76]  Herbert Dawid,et al.  EURACE: A massively parallel agent-based model of the European economy , 2008, Appl. Math. Comput..