Automatic Algorithm Selection for Complex Simulation Problems

To select the most suitable simulation algorithm for a given task is often difficult. This is due to intricate interactions between model features, implementation details, and runtime environment, which may strongly affect the overall performance. An automated selection of simulation algorithms supports users in setting up simulation experiments without demanding expert knowledge on simulation. Roland Ewald analyzes and discusses existing approaches to solve the algorithm selection problem in the context of simulation. He introduces a framework for automatic simulation algorithm selection and describes its integration into the open-source modelling and simulation framework James II. Its selection mechanisms are able to cope with three situations: no prior knowledge is available, the impact of problem features on simulator performance is unknown, and a relationship between problem features and algorithm performance can be established empirically. The author concludes with an experimental evaluation of the developed methods.

[1]  Neil Postman,et al.  Informing Ourselves to Death , 2013 .

[2]  Adelinde M. Uhrmacher,et al.  A plug-in-based architecture for random number generation in simulation systems , 2008, 2008 Winter Simulation Conference.

[3]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[4]  Brian H. Larose The development and implementation of a performance database server , 1993, CS / Technical report / Knoxville / University of Tennessee / Computer Science Department.

[5]  Hilan Bensusan,et al.  Meta-Learning by Landmarking Various Learning Algorithms , 2000, ICML.

[6]  Steven G. Johnson,et al.  FFTW: an adaptive software architecture for the FFT , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[7]  RICK SIOW MONG GOH,et al.  MLIST : AN EFFICIENT PENDING EVENT SET STRUCTURE FOR DISCRETE EVENT SIMULATION , 2004 .

[8]  Michael A. Gibson,et al.  Efficient Exact Stochastic Simulation of Chemical Systems with Many Species and Many Channels , 2000 .

[9]  Felipe Cucker,et al.  Learning Theory: An Approximation Theory Viewpoint: Index , 2007 .

[10]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[11]  Carl Tropper,et al.  On Determining How Many Computers to Use in Parallel VLSI Simulation , 2009, 2009 ACM/IEEE/SCS 23rd Workshop on Principles of Advanced and Distributed Simulation.

[12]  P. L'Ecuyer,et al.  Panel: strategic directions in simulation research , 1999, WSC'99. 1999 Winter Simulation Conference Proceedings. 'Simulation - A Bridge to the Future' (Cat. No.99CH37038).

[13]  Jan van Leeuwen,et al.  The Turing machine paradigm in contemporary computing , 2001 .

[14]  Corrado Priami,et al.  Stochastic pi-Calculus , 1995, Comput. J..

[15]  H. Kitano Systems Biology: A Brief Overview , 2002, Science.

[16]  Stephen John Turner,et al.  Optimistic protocol analysis in a performance analyser and prediction tool , 2005, Workshop on Principles of Advanced and Distributed Simulation (PADS'05).

[17]  N. Carr Is Google Making Us Stupid? , 2008, The Best Technology Writing 2009.

[18]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[19]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[20]  Adelinde M. Uhrmacher,et al.  SEQUENTIAL PROCESSING OF PDEVS MODELS , 2006 .

[21]  Lotfi A. Zadeh,et al.  Outline of a New Approach to the Analysis of Complex Systems and Decision Processes , 1973, IEEE Trans. Syst. Man Cybern..

[22]  Allen D. Malony,et al.  Design and implementation of a parallel performance data management framework , 2005, 2005 International Conference on Parallel Processing (ICPP'05).

[23]  F. Frank Chen,et al.  Parallel discrete event simulation of manufacturing systems: a technology survey , 1996 .

[24]  Frank Hampel,et al.  Robust statistics: a brief introduction and overview , 2001 .

[25]  H. Berendsen,et al.  COMPUTER-SIMULATION OF MOLECULAR-DYNAMICS - METHODOLOGY, APPLICATIONS, AND PERSPECTIVES IN CHEMISTRY , 1990 .

[26]  James Demmel,et al.  Statistical Models for Automatic Performance Tuning , 2001, International Conference on Computational Science.

[27]  W. Hsu,et al.  Algorithm selection for sorting and probabilistic inference: a machine learning-based approach , 2003 .

[28]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[29]  Michael Lees,et al.  An adaptive load management mechanism for distributed simulation of multi-agent systems , 2005, Ninth IEEE International Symposium on Distributed Simulation and Real-Time Applications.

[30]  Douglas W. Jones,et al.  An empirical comparison of priority-queue and event-set implementations , 1986, CACM.

[31]  Ian F. Akyildiz,et al.  Performance Analysis of Time Warp With Multiple Homogeneous Processors , 1991, IEEE Trans. Software Eng..

[32]  Bikramjit Banerjee,et al.  Advancing the Layered Approach to Agent-Based Crowd Simulation , 2008, 2008 22nd Workshop on Principles of Advanced and Distributed Simulation.

[33]  Kai Nagel,et al.  Using common graphics hardware for multi-agent traffic simulation with CUDA , 2009, SIMUTools 2009.

[34]  Naren Ramakrishnan,et al.  Note on generalization in experimental algorithmics , 2000, TOMS.

[35]  David A. Padua,et al.  A dynamically tuned sorting library , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[36]  S. Steinmetz,et al.  The American Heritage Dictionary of Science , 1986 .

[37]  Kurt Mehlhorn,et al.  Runtime prediction of real programs on real machines , 1997, SODA '97.

[38]  Wei Zhang,et al.  A Multi-State Q-Learning Approach for the Dynamic Load Balancing of Time Warp , 2010, 2010 IEEE Workshop on Principles of Advanced and Distributed Simulation.

[39]  John R. Rice,et al.  Future problem solving environments for computational science , 2000 .

[40]  Adelinde M. Uhrmacher,et al.  Discrete event modelling and simulation in systems biology , 2007, J. Simulation.

[41]  Stefan Leye,et al.  A Grid-Inspired Mechanism for Coarse-Grained Experiment Execution , 2008, 2008 12th IEEE/ACM International Symposium on Distributed Simulation and Real-Time Applications.

[42]  Bernard M. E. Moret,et al.  DIMACS Series in Discrete Mathematics and Theoretical Computer Science Towards a Discipline of Experimental Algorithmics , 2022 .

[43]  Fred W. Glover,et al.  Simulation optimization: a review, new developments, and applications , 2005, Proceedings of the Winter Simulation Conference, 2005..

[44]  James Demmel,et al.  Statistical Models for Empirical Search-Based Performance Tuning , 2004, Int. J. High Perform. Comput. Appl..

[45]  Michael Lees,et al.  Data access in distributed simulations of multi-agent systems , 2008, J. Syst. Softw..

[46]  Thomas M. Conte,et al.  Combining cluster sampling with single pass methods for efficient sampling regimen design , 2007, 2007 25th International Conference on Computer Design.

[47]  DE Economist A SURVEY ON THE BANDIT PROBLEM WITH SWITCHING COSTS , 2004 .

[48]  Pavel Pudlák Complexity Theory and Genetics: The Computational Power of Crossing Over , 2001, Inf. Comput..

[49]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[50]  William A. Goddard,et al.  Atomic-level simulation and modeling of biomacromolecules , 2001 .

[51]  Kendall Scott,et al.  UML distilled - applying the standard object modeling language , 1997 .

[52]  John A. Hamilton,et al.  Panel discussion: What makes good research in modeling and simulation: Assessing the quality, success, and utility of M&S research , 2008, 2008 Winter Simulation Conference.

[53]  David M. Nicol,et al.  Utility analysis of parallel simulation , 2003, Seventeenth Workshop on Parallel and Distributed Simulation, 2003. (PADS 2003). Proceedings..

[54]  Stephen John Turner,et al.  Distributed simulation performance data mining , 2001, Future Gener. Comput. Syst..

[55]  R. Bellman A Markovian Decision Process , 1957 .

[56]  Michael Lees,et al.  Decision-theoretic throttling for optimistic simulations of multi-agent systems , 2005, Ninth IEEE International Symposium on Distributed Simulation and Real-Time Applications.

[57]  Christopher D. Carothers,et al.  Scalable Time Warp on Blue Gene Supercomputers , 2009, 2009 ACM/IEEE/SCS 23rd Workshop on Principles of Advanced and Distributed Simulation.

[58]  Mary K. Vernon,et al.  Poems: end-to-end performance design of large parallel adaptive computational systems , 1998, WOSP '98.

[59]  Adelinde M. Uhrmacher,et al.  Automating the runtime performance evaluation of simulation algorithms , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[60]  R. Fujimoto Parallel and distributed simulation , 1995, Winter Simulation Conference Proceedings, 1995..

[61]  John R. Rice,et al.  PYTHIA: a knowledge-based system to select scientific algorithms , 1996, TOMS.

[62]  Werner Sandmann,et al.  Simultaneous Stochastic Simulation of Multiple Perturbations in Biological Network Models , 2007, CMSB.

[63]  Gordon S. Blair,et al.  Reflection, self-awareness and self-healing in OpenORB , 2002, WOSS '02.

[64]  A. L. Ruhkin Testing Randomness: A Suite of Statistical Procedures , 2001 .

[65]  William A. Wulf,et al.  A case against the GOTO , 1972, ACM '72.

[66]  Gabriel A. Wainer,et al.  Exploring Multi-Grained Parallelism in Compute-Intensive DEVS Simulations , 2010, 2010 IEEE Workshop on Principles of Advanced and Distributed Simulation.

[67]  Jürgen Schmidhuber,et al.  A Neural Network Model for Inter-problem Adaptive Online Time Allocation , 2005, ICANN.

[68]  Adelinde M. Uhrmacher,et al.  A Non-Fragmenting Partitioning Algorithm for Hierarchical Models , 2006, Proceedings of the 2006 Winter Simulation Conference.

[69]  A. Kerlavage,et al.  Complementary DNA sequencing: expressed sequence tags and human genome project , 1991, Science.

[70]  Shoji Takada,et al.  Competition between protein folding and aggregation with molecular chaperones in crowded solutions: insight from mesoscopic simulations. , 2003, Biophysical journal.

[71]  Catherine C. McGeoch Experimental analysis of algorithms , 1986 .

[72]  Nils J. Nilsson,et al.  MLC++, A Machine Learning Library in C++. , 1995 .

[73]  J. Brooks Why most published research findings are false: Ioannidis JP, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece , 2008 .

[74]  Yoav Shoham,et al.  Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions , 2002, CP.

[75]  Catherine C. McGeoch Experimental algorithmics , 2007, CACM.

[76]  John L. Klepeis,et al.  Anton, a special-purpose machine for molecular dynamics simulation , 2007, ISCA '07.

[77]  Alexander M. Millkey The Black Swan: The Impact of the Highly Improbable , 2009 .

[78]  Stefan Leye,et al.  A flexible architecture for performance experiments with the pi-Calculus and its extensions , 2010, SimuTools.

[79]  Malcolm Yoke-Hean Low Dynamic load-balancing for BSP Time Warp , 2002, Proceedings 35th Annual Simulation Symposium. SS 2002.

[80]  George F. Riley,et al.  Hardware Supported Time Synchronization in Multi-core Architectures , 2009, 2009 ACM/IEEE/SCS 23rd Workshop on Principles of Advanced and Distributed Simulation.

[81]  Andrea C. Arpaci-Dusseau,et al.  Towards realistic file-system benchmarks with CodeMRI , 2008, PERV.

[82]  R. Rosen Life Itself: A Comprehensive Inquiry Into the Nature, Origin, and Fabrication of Life , 1991 .

[83]  J. Elf,et al.  Spontaneous separation of bi-stable biochemical systems into spatial domains of opposite phases. , 2004, Systems biology.

[84]  Brad Calder,et al.  Automatically characterizing large scale program behavior , 2002, ASPLOS X.

[85]  Margaret Martonosi,et al.  Challenges in Computer Architecture Evaluation , 2003, Computer.

[86]  Andy Laws,et al.  From Wetware to Software: A Cybernetic Perspective of Self-adaptive Software , 2001, IWSAS.

[87]  Victor Eijkhout,et al.  Self-Adapting Numerical Software for Next Generation Applications , 2003, Int. J. High Perform. Comput. Appl..

[88]  Adelinde M. Uhrmacher,et al.  Multi-resolution spatial simulation for molecular crowding , 2008, 2008 Winter Simulation Conference.

[89]  Adelinde M. Uhrmacher,et al.  Introducing Variable Ports and Multi-Couplings for Cell Biological Modeling in DEVS , 2006, Proceedings of the 2006 Winter Simulation Conference.

[90]  Michail G. Lagoudakis,et al.  Algorithm Selection using Reinforcement Learning , 2000, ICML.

[91]  Franz Franchetti,et al.  On using ZENTURIO for performance and parameter studies on cluster and Grid architectures , 2003, Eleventh Euromicro Conference on Parallel, Distributed and Network-Based Processing, 2003. Proceedings..

[92]  Mark D. Hill,et al.  Amdahl's Law in the Multicore Era , 2008 .

[93]  Werner Sandmann,et al.  A Numerical Aggregation Algorithm for the Enzyme-Catalyzed Substrate Conversion , 2006, CMSB.

[94]  George L. Nemhauser,et al.  Handbooks in operations research and management science , 1989 .

[95]  Boyko Kakaradov Ultra-Fast Matrix Multiplication: An Empirical Analysis of Highly , 2004 .

[96]  Jack J. Dongarra,et al.  Automatically Tuned Linear Algebra Software , 1998, Proceedings of the IEEE/ACM SC98 Conference.

[97]  Krzysztof Pawlikowski,et al.  On credibility of simulation studies of telecommunication networks , 2002, IEEE Commun. Mag..

[98]  Eric R. Keiter,et al.  Redesigning the WARPED simulation kernel for analysis and application development , 2003, 36th Annual Simulation Symposium, 2003..

[99]  Peter A. Vanrolleghem,et al.  Intelligent configuration of numerical solvers of environmental ODE/DAE models using machine learning techniques , 2006 .

[100]  Georgios Theodoropoulos,et al.  Adaptive Support of Range Queries via Push-Pull Algorithms , 2007, 21st International Workshop on Principles of Advanced and Distributed Simulation (PADS'07).

[101]  Francesco Quaglia Software Diversity-Based Active Replication as an Approach for Enhancing the Performance of Advanced Simulation Systems , 2007, Int. J. Found. Comput. Sci..

[102]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[103]  Masaru Tomita,et al.  A multi-algorithm, multi-timescale method for cell simulation , 2004, Bioinform..

[104]  Adelinde M. Uhrmacher,et al.  Parallel and Distributed Spatial Simulation of Chemical Reactions , 2008, 2008 22nd Workshop on Principles of Advanced and Distributed Simulation.

[105]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[106]  Bart Selman,et al.  Algorithm portfolios , 2001, Artif. Intell..

[107]  Dennis Gannon,et al.  Developing component architectures for distributed scientific problem solving , 1998 .

[108]  A. Lo,et al.  Stock Market Prices Do Not Follow Random Walks: Evidence from a Simple Specification Test , 1987 .

[109]  Richard M. Fujimoto,et al.  Cloning parallel simulations , 2001, TOMC.

[110]  George Marsaglia,et al.  Seeds for random number generators , 2003, CACM.

[111]  Peter C. Cheeseman,et al.  Where the Really Hard Problems Are , 1991, IJCAI.

[112]  George Karypis,et al.  Multilevel algorithms for partitioning power-law graphs , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[113]  Andy Laws,et al.  Genetically Modified Software: Realizing Viable Autonomic Agency , 2005, WRAC.

[114]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[115]  Matteo Gagliolo,et al.  Towards Life-Long Meta Learning , 2005 .

[116]  J. R. Quinlan Learning With Continuous Classes , 1992 .

[117]  G. Marsaglia Random numbers fall mainly in the planes. , 1968, Proceedings of the National Academy of Sciences of the United States of America.

[118]  Tom DeMarco Software Engineering: An Idea Whose Time Has Come and Gone? , 2009, IEEE Software.

[119]  David M. Nicol Scalability, locality, partitioning and synchronization PDES , 1998, Workshop on Parallel and Distributed Simulation.

[120]  Brian Beckman,et al.  Time warp operating system , 1987, SOSP '87.

[121]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[122]  David S. Johnson,et al.  A theoretician's guide to the experimental analysis of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[123]  Mathias John,et al.  A Spatial Extension to the pi Calculus , 2008, Electron. Notes Theor. Comput. Sci..

[124]  Stefan Leye,et al.  One Modelling Formalism & Simulator Is Not Enough! A Perspective for Computational Biology Based on James II , 2008, FMSB.

[125]  Roland Ewald,et al.  Simulation of load balancing algorithms for discrete event simulations , 2006 .

[126]  E. D. Schneider,et al.  Life as a manifestation of the second law of thermodynamics , 1994 .

[127]  Adelinde M. Uhrmacher,et al.  A Simulation Approach to Facilitate Parallel and Distributed Discrete-Event Simulator Development , 2006, 2006 Tenth IEEE International Symposium on Distributed Simulation and Real-Time Applications.

[128]  Yoav Shoham,et al.  Empirical hardness models: Methodology and a case study on combinatorial auctions , 2009, JACM.

[129]  Richard E. Ladner,et al.  The influence of caches on the performance of sorting , 1997, SODA '97.

[130]  R. Lewontin ‘The Selfish Gene’ , 1977, Nature.

[131]  Sudip K. Seal,et al.  GPU-based Real-Time Execution of Vehicular Mobility Models in Large-Scale Road Network Scenarios , 2009, 2009 ACM/IEEE/SCS 23rd Workshop on Principles of Advanced and Distributed Simulation.

[132]  Hong Li,et al.  Efficient formulation of the stochastic simulation algorithm for chemically reacting systems. , 2004, The Journal of chemical physics.

[133]  Adelinde M. Uhrmacher,et al.  A flexible and scalable experimentation layer , 2008, 2008 Winter Simulation Conference.

[134]  Bruce Edmonds,et al.  What is Complexity? - The philosophy of complexity per se with application to some examples in evolution , 1995 .

[135]  Sheldon M. Ross,et al.  Introduction to probability models , 1975 .

[136]  Makoto Matsumoto,et al.  Common defects in initialization of pseudorandom number generators , 2007, TOMC.

[137]  Richard M. Fujimoto,et al.  GTW: a time warp system for shared memory multiprocessors , 1994, Proceedings of Winter Simulation Conference.

[138]  Alex M. Andrew,et al.  ROBOT LEARNING, edited by Jonathan H. Connell and Sridhar Mahadevan, Kluwer, Boston, 1993/1997, xii+240 pp., ISBN 0-7923-9365-1 (Hardback, 218.00 Guilders, $120.00, £89.95). , 1999, Robotica (Cambridge. Print).

[139]  Jun Wang,et al.  Optimizing time warp simulation with reinforcement learning techniques , 2007, 2007 Winter Simulation Conference.

[140]  Adelinde M. Uhrmacher,et al.  A parallel and distributed discrete event approach for spatial cell-biological simulations , 2008, PERV.

[141]  Hugh E. Williams,et al.  Managing and using MySQL , 2002 .

[142]  Y. Shoham,et al.  SATzilla : An Algorithm Portfolio for SAT ∗ , 2004 .

[143]  Ivana Kruijff-Korbayová,et al.  A Portfolio Approach to Algorithm Selection , 2003, IJCAI.

[144]  D.M. Nicol,et al.  Performance modeling of the IDES framework , 1997, Proceedings 11th Workshop on Parallel and Distributed Simulation.

[145]  Toby Walsh,et al.  How Not To Do It , 1995 .

[146]  Alan Edelman,et al.  PetaBricks: a language and compiler for algorithmic choice , 2009, PLDI '09.

[147]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[148]  Ian T. Foster,et al.  Globus: a Metacomputing Infrastructure Toolkit , 1997, Int. J. High Perform. Comput. Appl..

[149]  Stephen John Turner,et al.  Alternative Solutions for Distributed Simulation Cloning , 2003, Simul..

[150]  T. Wilding Using genetic algorithms to construct portfolios , 2003 .

[151]  Donald E. Knuth,et al.  The Art of Computer Programming: Volume 3: Sorting and Searching , 1998 .

[152]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, WSC 2008.

[153]  Takuji Nishimura,et al.  Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator , 1998, TOMC.

[154]  Guanhua Yan,et al.  Simulation of large scale networks using SSF , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[155]  Jayadev Misra,et al.  Distributed discrete-event simulation , 1986, CSUR.

[156]  Danny Weyns,et al.  Anticipatory Vehicle Routing using Delegate Multi-Agent Systems , 2007, 2007 IEEE Intelligent Transportation Systems Conference.

[157]  Donald E. Knuth,et al.  Big Omicron and big Omega and big Theta , 1976, SIGA.

[158]  L. You,et al.  Stochastic vs. deterministic modeling of intracellular viral kinetics. , 2002, Journal of theoretical biology.

[159]  Gautam Mitra,et al.  A review of portfolio planning: Models and systems , 2003 .

[160]  Naren Ramakrishnan,et al.  MyPYTHIA: a recommendation portal for scientific software and services , 2002, Concurr. Comput. Pract. Exp..

[161]  Tad Hogg,et al.  Phase Transitions and the Search Problem , 1996, Artif. Intell..

[162]  Peter Fritzson,et al.  A Generalized Framework for Abstraction and Dynamic Loading of Numerical Solvers , 2006 .

[163]  Nael B. Abu-Ghazaleh,et al.  A framework for performance analysis of parallel discrete event simulators , 1997, WSC '97.

[164]  T. L. Lai Andherbertrobbins Asymptotically Efficient Adaptive Allocation Rules , 1985 .

[165]  Luciano Bononi,et al.  Concurrent replication of parallel and distributed simulations , 2005, Workshop on Principles of Advanced and Distributed Simulation (PADS'05).

[166]  Adelinde M. Uhrmacher,et al.  Data mining for simulation algorithm selection , 2009, SIMUTools 2009.

[167]  Steffen Straßburger,et al.  Scalability in distributed simulations of agent-based models , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[168]  Daniel T Gillespie,et al.  Stochastic simulation of chemical kinetics. , 2007, Annual review of physical chemistry.

[169]  Allen D. Malony,et al.  PerfExplorer: A Performance Data Mining Framework For Large-Scale Parallel Computing , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[170]  A. G. Malliaris,et al.  Chapter 1 Portfolio theory , 1995, Finance.

[171]  Jun Wang,et al.  Using genetic algorithms to limit the optimism in Time Warp , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[172]  Noel A Cressie,et al.  Statistics for Spatial Data. , 1992 .

[173]  Scott R. Kohn,et al.  Toward a Common Component Architecture for High-Performance Scientific Computing , 1999, HPDC.

[174]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[175]  Miodrag Potkonjak,et al.  Algorithm Selection: A Quantitative Computation-intensive Optimization Approach , 1994, IEEE/ACM International Conference on Computer-Aided Design.

[176]  Michael Mascagni,et al.  Testing parallel random number generators , 2003, Parallel Comput..

[177]  D. Deutsch Quantum theory, the Church–Turing principle and the universal quantum computer , 1985, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences.

[178]  Karsten Weihe On the Differences between "Practical" and "Applied" , 2000, Algorithm Engineering.

[179]  Jürgen Schmidhuber,et al.  Adaptive Online Time Allocation to Search Algorithms , 2004, ECML.

[180]  Michael Mascagni,et al.  Parameterizing parallel multiplicative lagged-Fibonacci generators , 2004, Parallel Comput..

[181]  Gabor Karsai,et al.  An Approach to Self-adaptive Software Based on Supervisory Control , 2001, IWSAS.

[182]  D. Noble Music of life : biology beyond the genome , 2006 .

[183]  Ronald L. Rivest,et al.  Introduction to Algorithms, Second Edition , 2001 .

[184]  Jussi Rintanen Phase Transitions in Classical Planning: An Experimental Study , 2004, ICAPS.

[185]  Stefan Jähnichen,et al.  TOWARDS AN ARCHITECTURE FOR SIMULATION ENVIRONMENTS , 2008 .

[186]  Paul Bratley,et al.  A guide to simulation , 1983 .

[187]  Gabriel A. Wainer,et al.  DEVStone: a benchmarking technique for studying performance of DEVS modeling and simulation environments , 2005, Ninth IEEE International Symposium on Distributed Simulation and Real-Time Applications.

[188]  Michael R. Fellows,et al.  Parameterized Complexity: The Main Ideas and Some Research Frontiers , 2009, ISAAC.

[189]  B. Zeigler,et al.  DEVS / RMI — AnAuto-Adaptive and Reconfigurable Distributed Simulation Environment for Engineering Studies , 2005 .

[190]  Mehryar Mohri,et al.  Multi-armed Bandit Algorithms and Empirical Evaluation , 2005, ECML.

[191]  K. Mani Chandy,et al.  Distributed Simulation: A Case Study in Design and Verification of Distributed Programs , 1979, IEEE Transactions on Software Engineering.

[192]  James Demmel,et al.  Statistical Modeling of Feedback Data in an Automatic Tuning System , 2000 .

[193]  Hod Lipson,et al.  Distilling Free-Form Natural Laws from Experimental Data , 2009, Science.

[194]  Mathias John,et al.  Combining micro and macro-modeling in DEVS for computational biology , 2007, 2007 Winter Simulation Conference.

[195]  Nicolò Cesa-Bianchi,et al.  Gambling in a rigged casino: The adversarial multi-armed bandit problem , 1995, Proceedings of IEEE 36th Annual Foundations of Computer Science.

[196]  Arthur E. Kirkpatrick,et al.  Assessing open source software as a scholarly contribution , 2009, Commun. ACM.

[197]  William H. Hsu,et al.  A machine learning approach to algorithm selection for $\mathcal{NP}$ -hard optimization problems: a case study on the MPE problem , 2007, Ann. Oper. Res..

[198]  Dror Rawitz,et al.  The hardness of cache conscious data placement , 2002, POPL '02.

[199]  Jürgen Schmidhuber,et al.  Learning dynamic algorithm portfolios , 2006, Annals of Mathematics and Artificial Intelligence.

[200]  Jeffrey O. Kephart,et al.  The Vision of Autonomic Computing , 2003, Computer.

[201]  Brian Logan,et al.  The distributed simulation of multiagent systems , 2001, Proc. IEEE.

[202]  A. Turing On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[203]  Thomas M. Cover,et al.  Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) , 2006 .

[204]  Vittorio Cortellessa,et al.  AN ANALYSIS OF THE EFFICIENCY OFOPTIMISTICALLY SYNCHRONIZED PARALLEL SIMULATORS , 2007 .

[205]  Nicolas Le Novère,et al.  SED-ML - An XML Format for the Implementation of the MIASE Guidelines , 2008, CMSB.

[206]  Nicolas Le Novère,et al.  Particle-Based Stochastic Simulation in Systems Biology , 2006 .

[207]  B. Logan,et al.  The Distributed Simulation of Multi-Agent Systems , 2000 .

[208]  Jack L. Treynor,et al.  MUTUAL FUND PERFORMANCE* , 2007 .

[209]  François E. Cellier,et al.  Continuous System Simulation , 2006 .

[210]  David Maxwell Chickering,et al.  A Bayesian Approach to Tackling Hard Computational Problems (Preliminary Report) , 2001, Electron. Notes Discret. Math..

[211]  Adelinde M. Uhrmacher,et al.  An Algorithm Selection Approach for Simulation Systems , 2008, 2008 22nd Workshop on Principles of Advanced and Distributed Simulation.

[212]  Leslie Pack Kaelbling,et al.  Learning in embedded systems , 1993 .

[213]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[214]  R. Laddaga Creating robust software through self-adaptation , 1999, IEEE Intelligent Systems and their Applications.

[215]  Karl-Georg Steffens The history of approximation theory : from Euler to Bernstein , 2006 .

[216]  Tianhai Tian,et al.  A multi-scaled approach for simulating chemical reaction systems. , 2004, Progress in biophysics and molecular biology.

[217]  João Gama,et al.  On Data and Algorithms: Understanding Inductive Performance , 2004, Machine Learning.

[218]  Francis Heylighen,et al.  Principles of Systems and Cybernetics: an evolutionary perspective , 1991 .

[219]  Giuseppe Iazeolla,et al.  A Methodology to Predict the Performance of Distributed Simulations , 2010, 2010 IEEE Workshop on Principles of Advanced and Distributed Simulation.

[220]  Adelinde M. Uhrmacher,et al.  The event queue problem and PDevs , 2007, SpringSim '07.

[221]  Roger W. Hockney A framework for benchmark performance analysis , 1991 .

[222]  Michail G. Lagoudakis,et al.  Selecting the Right Algorithm , 2001 .

[223]  Sathish S. Vadhiyar,et al.  Automatically Tuned Collective Communications , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[224]  David S. Johnson,et al.  Some simplified NP-complete problems , 1974, STOC '74.

[225]  Vlatka Hlupic Discrete-Event Simulation Software: What the Users Want , 1999, Simul..

[226]  J. Bather,et al.  Multi‐Armed Bandit Allocation Indices , 1990 .

[227]  Pierre L'Ecuyer,et al.  Random numbers for simulation , 1990, CACM.

[228]  Christopher Small,et al.  Does Systems Research Measure Up , 1997 .

[229]  Robert L. Henderson,et al.  Job Scheduling Under the Portable Batch System , 1995, JSSPP.

[230]  J. Gentle Numerical Linear Algebra for Applications in Statistics , 1998 .

[231]  H. Robbins Some aspects of the sequential design of experiments , 1952 .

[232]  Adelinde M. Uhrmacher,et al.  A component-based simulation layer for JAMES , 2004, 18th Workshop on Parallel and Distributed Simulation, 2004. PADS 2004..

[233]  Adelinde M. Uhrmacher,et al.  Plug'n Simulate , 2007, 40th Annual Simulation Symposium (ANSS'07).

[234]  Pierre L'Ecuyer,et al.  TestU01: A C library for empirical testing of random number generators , 2006, TOMS.

[235]  Olivier Dalle,et al.  Design considerations for M&S software , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[236]  Richard M. Fujimoto Parallel simulation: distributed simulation systems , 2003, WSC '03.

[237]  J. Weizenbaum Computer Power And Human Reason: From Judgement To Calculation , 1978 .

[238]  Albert-László Barabási,et al.  Linked - how everything is connected to everything else and what it means for business, science, and everyday life , 2003 .

[239]  Samir Ranjan Das Adaptive protocols for parallel discrete event simulation , 1996, Winter Simulation Conference.

[240]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[241]  David M. Nicol,et al.  Performance prediction of a parallel simulator , 1999, Proceedings Thirteenth Workshop on Parallel and Distributed Simulation. PADS 99. (Cat. No.PR00155).

[242]  James R Wilson,et al.  Responsible authorship and peer review , 2002, Science and engineering ethics.

[243]  R. Plackett,et al.  THE DESIGN OF OPTIMUM MULTIFACTORIAL EXPERIMENTS , 1946 .

[244]  Thomas Stützle,et al.  SATLIB: An Online Resource for Research on SAT , 2000 .

[245]  Robert Almeder,et al.  Pragmatism and Science , 2013 .

[246]  W. R. Thompson ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES , 1933 .

[247]  D. Gillespie Exact Stochastic Simulation of Coupled Chemical Reactions , 1977 .

[248]  Sugih Jamin,et al.  An Efficient Synchronization Mechanism for Mirrored Game Architectures , 2002, NetGames '02.

[249]  Michael Lees,et al.  Simulation Engines for Multi-Agent Systems , 2009, Multi-Agent Systems.

[250]  John R. Koza,et al.  Hidden Order: How Adaptation Builds Complexity. , 1995, Artificial Life.

[251]  Yoav Shoham,et al.  Boosting as a Metaphor for Algorithm Design , 2003, CP.

[252]  Jürgen Schmidhuber,et al.  Dynamic Algorithm Portfolios , 2006, AI&M.

[253]  Peter Grassberger,et al.  On correlations in “good” random number generators , 1993 .

[254]  B. P. Ziegler,et al.  Theory of Modeling and Simulation , 1976 .

[255]  Philippe Jorion,et al.  Portfolio Optimization in Practice , 1992 .

[256]  Stefan Leye,et al.  Performance Issues in Evaluating Models and Designing Simulation Algorithms , 2009, 2009 International Workshop on High Performance Computational Systems Biology.

[257]  B. Segal,et al.  Grid computing: the European Data Grid Project , 2000, 2000 IEEE Nuclear Science Symposium. Conference Record (Cat. No.00CH37149).

[258]  Roland Ewald,et al.  Modeling, Simulation and Games , 2008, Mensch & Computer Workshopband.

[259]  Naren Ramakrishnan,et al.  PYTHIA-II: a knowledge/database system for managing performance data and recommending scientific software , 2000, TOMS.

[260]  P. Hellekalek Good random number generators are (not so) easy to find , 1998 .

[261]  Adelinde M. Uhrmacher,et al.  Selecting Simulation Algorithm Portfolios by Genetic Algorithms , 2010, 2010 IEEE Workshop on Principles of Advanced and Distributed Simulation.

[262]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[263]  Sally A. McKee,et al.  Efficient architectural design space exploration via predictive modeling , 2008, TACO.

[264]  Philip A. Wilsey,et al.  WARPED: a time warp simulation kernel for analysis and application development , 1996, Proceedings of HICSS-29: 29th Hawaii International Conference on System Sciences.

[265]  Jan Himmelspach Konzeption, Realisierung und Verwendung eines allgemeinen Modellierungs-, Simulations- und Experimentiersystems , 2007 .

[266]  Lawrence Rauchwerger,et al.  An Adaptive Algorithm Selection Framework , 2004, IEEE PACT.

[267]  Adelinde M. Uhrmacher,et al.  Experiments with Single Core, Multi-core, and GPU Based Computation of Cellular Automata , 2009, 2009 First International Conference on Advances in System Simulation.

[268]  Bart Selman,et al.  Algorithm Portfolio Design: Theory vs. Practice , 1997, UAI.

[269]  Stefan Leye,et al.  Flexible experimentation in the modeling and simulation framework JAMES II - implications for computational systems biology , 2010, Briefings Bioinform..

[270]  Tad Hogg,et al.  An Economics Approach to Hard Computational Problems , 1997, Science.

[271]  S. Turner,et al.  A PERFORMANCE ANALYSER AND PREDICTION TOOL FOR PARALLEL DISCRETE EVENT SIMULATION , 2003 .

[272]  Georgios Theodoropoulos,et al.  TIME WINDOWS IN MULTI-AGENT DISTRIBUTED SIMULATION , 2004 .

[273]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[274]  Bing Wang,et al.  Experimental analysis of logical process simulation algorithms in JAMES II , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[275]  Roger D. Chamberlain,et al.  Evaluating the use of pre-simulation in VLSI circuit partitioning , 1994, PADS '94.

[276]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[277]  Marek Petrik,et al.  Statistically Optimal Combination of Algorithms , 2004 .

[278]  Philip Heidelberger,et al.  Computer Performance Evaluation Methodology , 1984, IEEE Transactions on Computers.

[279]  E. Dijkstra On the Role of Scientific Thought , 1982 .

[280]  Xingfu Wu,et al.  Prophesy: an infrastructure for performance analysis and modeling of parallel and grid applications , 2003, PERV.

[281]  Stefan Leye,et al.  An Efficient and Adaptive Mechanism for Parallel Simulation Replication , 2009, 2009 ACM/IEEE/SCS 23rd Workshop on Principles of Advanced and Distributed Simulation.

[282]  Eric A. Brewer,et al.  High-level optimization via automated statistical modeling , 1995, PPOPP '95.

[283]  Reinhart Heinrich,et al.  The Roles of APC and Axin Derived from Experimental and Theoretical Analysis of the Wnt Pathway , 2003, PLoS biology.

[284]  Santosh Pande,et al.  Performance prediction of large-scale parallel discrete event models of physical systems , 2005, Proceedings of the Winter Simulation Conference, 2005..

[285]  Catherine C. McGeoch Analyzing algorithms by simulation: variance reduction techniques and simulation speedups , 1992, CSUR.

[286]  Xiaoming Li,et al.  Optimizing Matrix Multiplication with a Classifier Learning System , 2005, LCPC.

[287]  Rajkumar Buyya,et al.  GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing , 2002, Concurr. Comput. Pract. Exp..

[288]  Michael Lees,et al.  Performance Analysis of Shared Data Access Algorithms for Distributed Simulation of Multi-Agent Systems , 2006, 20th Workshop on Principles of Advanced and Distributed Simulation (PADS'06).

[289]  Roland Ewald,et al.  Large-Scale Design Space Exploration of SSA , 2008, CMSB.

[290]  Linda R Petzold,et al.  Efficient step size selection for the tau-leaping simulation method. , 2006, The Journal of chemical physics.

[291]  Francesco Quaglia A Middleware Level Active Replication Manager for High Performance HLA-based Simulations on SMP Systems , 2006, 2006 Tenth IEEE International Symposium on Distributed Simulation and Real-Time Applications.

[292]  Eugene Fink,et al.  How to Solve It Automatically: Selection Among Problem Solving Methods , 1998, AIPS.

[293]  D. Gillespie A rigorous derivation of the chemical master equation , 1992 .

[294]  D. Gillespie A General Method for Numerically Simulating the Stochastic Time Evolution of Coupled Chemical Reactions , 1976 .

[295]  Philip Heidelberger Statistical analysis of parallel simulations , 1986, WSC '86.