Statistical runtime verification of agent-based simulations

As a consequence of the growing adoption of agent-based simulations as decision making tools in various (potentially also critical) areas, questions of veracity and validity become increasingly important. In general software and hardware development, formal verification – particularly model checking – has been applied successfully to a wide range of problems; due to their immense complexity, however, agent-based simulations lend themselves to conventional formal verification only in very simple cases and at a disproportionately high cost. The purpose of this work is to address this problem and present a statistical runtime verification approach which focusses on the analysis of the temporal behaviour of large-scale probabilistic agent-based simulations. The approach is tailored to the particular mix of characteristics that agent-based simulations typically exhibit: large populations, randomness, heterogeneity, temporal boundedness and the existence of multiple observational levels. It combines the ideas of runtime verification and statistical model checking and allows for the temporal verification of simulations with hundreds or thousands of constituents and probabilistic state transitions. Instead of requiring a formal model, verification is performed upon traces of the original simulation obtained through repeated execution. Properties are checked on-the-fly, i.e. during the execution of the simulation, which is achieved by interleaving simulation and verification. Evaluation is lazy, i.e. a simulation step is performed only if the property has not already been satisfied or refuted. This reduces the amount of simulation to a minimum and restricts state space exploration to the smallest fragment necessary for finding a definite answer to the given property. Verification results are approximate, but the precision is clearly quantifiable and adjustable by varying the number of simulation runs.