Benchmark: A Nonlinear Reachability Analysis Test Set from Numerical Analysis

The field of numerical analysis has developed numerous benchmarks for evaluating differential and algebraic equation solvers. In this paper, we describe a set of benchmarks commonly used in numerical analysis that may also be effective for evaluating continuous and hybrid systems reachability and verification methods. Many of these examples are challenging and have highly nonlinear differential equations and upwards of tens of dimensions (state variables). Additionally, many examples in numerical analysis are originally encoded as differential algebraic equations (DAEs) with index greater than one or as implicit differential equations (IDEs), which are challenging to model as hybrid automata. We present executable models for ten benchmarks from a test set for initial value problems (IVPs) in the SpaceEx format (allowing for nonlinear equations instead of restricting to affine) and illustrate their conversion to several other formats (dReach, Flow*, and the MathWorks Simulink/Stateflow [SLSF]) using the HyST tool. For some instances, we present successful analysis results using dReach, Flow*, and SLSF. Category: academic Difficulty: low through challenge 1 Context and Origins Verification and validation are important tasks that are applied broadly in many fields in recent years such as embedded systems, power electronics, networked control systems, and aerospace systems [4, 16, 17]. Many different verification methods and tools have been developed for reachability analysis of hybrid systems [2, 3, 7, 14]. The challenges in verification of continuous and hybrid systems are many, and include for example complex nonlinear dynamics, highdimensional state-spaces, and bounded vs. unbounded time. To evaluate novel verification methods and tools, we need to evaluate and test them using a variety of diverse benchmarks, that are ideally standardized. However, these benchmarks are not standardized, so it is difficult to evaluate whether particular state representations (e.g., zonotopes [1], Taylor models [7], support functions [13], polyhedra, hypercubes [5], symbolic/SMT formulas [14], etc.) and verification techniques are superior for different classes of hybrid automata. In this paper, we present a set of ten different, executable benchmarks to aid in the development of a standardized set of benchmarks for the verification community to evaluate verification methods and tools. These benchmarks are derived from a test set for initial value problem (IVP) G.Frehse and M.Althoff (eds.), ARCH15 (EPiC Series in Computer Science, vol. 34), pp. 89–97 89 Benchmark: A Nonlinear Reachability Analysis Test Set from Numerical Analysis Tran, Nguyen, and Johnson solvers from numerical analysis [18,19], and include systems modeled by nonlinear ordinary differential equations (ODEs) and differential-algebraic equations (DAEs). While some of these benchmarks are standard and well-known in the hybrid systems verification community (e.g., the Van der Pol oscillator), the majority (to the best of the authors’ knowledge) have not previously been considered for benchmarking hybrid systems reachability tools, although similar recent initiatives are ongoing [8]. This test set and others are used as standardized benchmarks to compare IVP solvers and helped move numerical analysis into the mainstream [10, 11, 15], and this was one of the original goals in developing standardized benchmarks: “The problems, methods and comparison criteria are specified very carefully. One objective in doing so is to provide a rigorous conceptual basis for comparing methods. Another is to provide a useful standard for such comparisons” [15]. We then hope that the development of similar standard benchmark sets for verification purposes will help move reachability analysis and verification into industrial adoption. The benchmarks are shown in Table 2.2 and come from a variety of fields, including biology, environmental science, celestial mechanics, chemistry, and electronics. The benchmarks are independent from a specific approach to evaluate reachability algorithms, as they have different classes of nonlinear dynamics, dimensionality, etc. All the benchmarks are purely continuous, and do not include any hybrid or switched behavior. Most of them contain highly stiff nonlinear differential equations and a high number of state variables, which make them challenging to analyze with existing techniques and tools, but they may serve as benchmarks to evaluate the next generation of techniques and tools. All the benchmarks are IVPs and specify (1) the initial conditions, (2) the ODEs or DAEs, (3) a final state, and (4) the time to reach the final state; see Table A.1. For reachability analysis and safety verification, the final state—or actually a neighborhood about the final state to avoid minor numerical issues—specifies the bad set of states, and reachability from the initial state to the final state may be checked (either using the time or not). All of benchmarks are first described in the input format for SpaceEx1, and are then translated to other formats including dReach, Flow*, and Matlab Simulink/Stateflow (SLSF) using the HyST model transformation tool [6]. To validate the conversion of the benchmarks from their original descriptions (from the paper describing them [18] and some Fortran code [19]) to hybrid automata, simulations in Matlab were conducted of equivalent continuoustime SLSF charts (generated using HyST), and compared to existing simulation results [18,19]. Additionally, several benchmarks are analyzed using Flow* and dReach. The problems are diverse, with the number of state variables varying from two to twenty eight, so these benchmarks may be useful to evaluate reachability algorithms, verification methods, and tools. 2 Brief Descriptions For brevity, we do not describe in detail all the benchmarks in Table 2.2, but refer to their origins and detailed mathematical definitions in [18, 19].2 We provide executable models (as SpaceEx hybrid automata, SLSF, and the other formats supported by HyST) for all the benchmarks in the supplementary material. In this section, we focus on presenting the Chemical Akzo Nobel problem taken from the test set in [18,19] as it is a nonlinear DAE systems and is nontrivial to model as hybrid automata. 1This allows for nonlinear functions, which may be specified and parsed by SpaceEx, but not analyzed as only affine functions are supported. 2The executable models are included on the ARCH website and are also available online from the HyST website at: http://verivital.com/hyst/.