Invariance in Stochastic Dynamical Control Systems

Aim of this paper is to set invariance in stochastic dynamical control systems. Given a set within which the state of the dynamical system should evolve, we study conditions for finding a control strategy that maximizes the probability for the state to be in the given set within a fixed a–priori finite time horizon. We formulate an optimal control problem and we solve the problem at hand by using a dynamic programming approach. Some results towards a generalization of these results to the case of infinite–time horizon are also derived. Keywords—Stochastic dynamical control system, Stochastic Invariance Problem, Dynamic Programming, Optimal Control. I. I NTRODUCTION In the last few years, several stochastic hybrid models have been proposed and studied in the literature (see [15], [4] for an overview), because of their capability to capture non–smooth phenomena arising in real life complex systems such as Air Traffic Management (ATM) [4]. A general model of stochastic hybrid systems has been presented in [6], which includes as special cases many stochastic hybrid systems available in the literature as for example Piecewise Deterministic Markov Processes [7], Switching Diffusion Processes [9] and Stochastic Hybrid Systems [10] (see [15] for a formal comparison on these models). However, the analysis of the dynamical properties of these sophisticated models is very difficult. Preliminary work on the analysis of stochastic hybrid system properties was given in [5] and [6], where reachability problems of Piecewise Deterministic Markov Processes and of General Stochastic Hybrid Systems were formulated. Reachability problems for stochastic dynamical systems arise for example in a conflict resolution problem in afree flight configuration, one of the most challenging problems in the context of ATM (see e.g. [11]). Unfortunately, results reported in [5] and [6] do not include computationally feasible methodologies for approaching this kind of reachability problem. In [13], the conflict resolution problem is addressed with a Monte Carlo simulation based approach. In [1], some reachability problems for discrete–time stochastic hybrid systems are considered and solved. This work has been partially supported by the HYCON Network of Excellence, contract number FP6-IST-511368 and by Ministero dell’Istruzione, dell’Universita’ e della Ricerca under Projects MACSI and SCEF (PRIN05). G. Pola and M.D. Di Benedetto are with the Department of Electrical Engineering and Computer Science, Center of Excellence DEWS, University of L’Aquila, Poggio di Roio, 67040 L’Aquila, Italy, {pola,dibenede }@ing.univaq.it J. Lygeros is with the Department of Electrical Engineering, University of Patras, GR26500, Patras, Greece, lygeros@ee.upatras.gr In this paper, we present a first contribution towards the development of a computationally feasible solution to reachability problems for stochastic dynamical control systems. We consider non–linear discrete–time stochastic dynamical control systems and formulate for this class of systems theStochastic Invariance Problem: Given a set, within which the state of the dynamical system must evolve, we derive conditions for finding a control strategy that maximizes the probability for the state to be in the given target set, within a fixed a–priori finite time horizon. The general formulation of the Stochastic Invariance Problem provides a way for addressing reachability problems as a special case (i.e. finding an optimal control strategy guaranteeing to reach a target set at a given time instant with maximal probability). Moreover, some new definitions of invariant and contractive sets are introduced, which generalize to the stochastic setting well–known notions available in the context of deterministic dynamical systems (e.g. [3]). To solve the Stochastic Invariance Problem, we use a dynamic programming approach [2]. In particular, we reformulate the Stochastic Invariance Problem as an optimal control problem and solve the optimal control problem using a dynamic programming approach. A generalization of the results in finite–time horizon to the case of infinite– time horizon is also described. A specialization of the results to the context of one dimensional stochastic control affine systems is presented as an example of application of the proposed methodology. The methodology proposed in this paper has been applied in [16] to a quantitative finance problem. In particular [16] sets a unified framework to treat optimal dynamic assets allocation, and generalizes classical Markowitz Portfolio approach [14] to multi–period investments and non– gaussian hypotheses on asset classes performance stochastic dynamics. This paper is organized as follows. Section 2 is devoted to the formal definition of the Stochastic Invariance Problem. In Section 3, we solve the Stochastic Invariance Problem. In Section 4 we address the special case of one dimensional stochastic control affine systems with gaussian noise. Section 5 offers some concluding remarks. II. PRELIMINARIES AND PROBLEM STATEMENT This section is devoted to the formal definition of the class of systems and of the control problem that we address in this paper. Consider the following non–linear stochastic discrete– time dynamical control system: xk+1 = f(xk, uk, Wk, k), (1) where, for anyk ∈ N: • xk ∈ X ⊂ R is the state andX is the state space; • uk ∈ U ⊂ R is the control input andU is the control input space; • Wk ∈ W ⊂ R is a random variable with probability density functionpWk , ∀k ∈ N and W is the noise space; • f : X× U ×W × N → X is the vector field. We assume that x0 is given by means of a random variableX0, whose probability density function is pX0 and that random variablesX0, Wk, ∀k ∈ N are independent one each other. Given any x ∈ X, u ∈ U and k ∈ N, the probability density function of random variable f(x, u, Wk, k) is denoted bypf(x,u,Wk,k). We focus on the following class of controls μ: U = {μ : X × N → U}, namely, the class of time–varying feedback functions. Let UN be the class of control inputs sequence π = {μk}k=0,1,...,N such thatμk ∈ U , for anyk = 0, 1, ..., N . Any π ∈ UN is calledcontrol policy. Given N ∈ N, and π = { μ0, μ1, ..., μN−1 } ∈ UN−1, we set π = { μk, μk+1, ..., μN−1 } , for any k ∈ N; thenπ = π. In safety critical problems (e.g. [8] and the references therein), which arise in many engineering domains as Air Traffic Management [13], finance [16], etc., an important goal is to find an optimal control strategy guaranteeing to reach a given target set or to remain in a given target set, within a finite time horizon. The notions of reachable and invariant sets are therefore very important (see e.g. [3]) and can be extended to stochastic dynamical control systems, as indicated below. Given a sequence of sets Σk ⊂ X, k = 0, 1, ... representing the sets of ‘good’ states within which the state evolution of system (1) must evolve, our problem is to find a control policy that maximizes the probability of the state xk to be inΣk, for any timek within a finite time horizon N , i.e. x(k) ∈ Σk, ∀k = 0, 1, ..., N. More formally, let (Ω,F , P ) be the probability space associated with system (1). Problem 1: (Stochastic Invariance Problem) Given a finite time horizon N ∈ N and a sequence of sets {Σk}k=0,1,...,N, where for anyk = 0, 1, ..., N , Σk are Borel subsets of X, find the optimal control policy π∗ ∈ UN−1 that maximizes P ({ω ∈ Ω : xk ∈ Σk, ∀k = 0, 1, ...,N}). (2) Denote byp∗(N) the optimal value attained by (2). For notational simplicity, we set P ({ω ∈ Ω : xk ∈ Σk, ∀k = 0, 1, ..., N}) = P (xk ∈ Σk, ∀k = 0, 1, ...,N). (3) The formulation of the Stochastic Invariance Problem is quite general to include as special cases also some other sub–problems as the classical reachability problem, invariance problem and the characterization of contractive sets in a stochastic fashion, thus generalizing classical notions, well–known in the context of deterministic systems (e.g. [3]) to the case of stochastic dynamical control systems, as shown in the following definition. Definition 1: Given N ∈ N and a control policyπ ∈ UN , a setΣ ⊂ X is said to be: • reachable in N steps with probabilityp ∈ [0, 1] if P (xN ∈ Σ, xN ∈ X, xN−1 ∈ X, ..., x0 ∈ X) = p; (4) • invariant in N steps with probabilityp ∈ [0, 1] if P (xk ∈ Σ, ∀k = 0, 1, ...,N) = p; (5) • λ–contractive in N steps with probabilityp ∈ [0, 1], for someλ ∈ (0, 1), if P (xk ∈ λΣ, ∀k = 0, 1, ...,N) = p. (6) A set Σ ⊂ X is said to beinvariant, with probability p if lim N→∞ P (xk ∈ Σ, ∀k = 0, 1, ...,N) = p andλ–contractive, with probabilityp if lim N→∞ P (xk ∈ λΣ, ∀k = 0, 1, ..., N) = p. It is readily seen that the tasks of finding optimal control policies maximizing probability quantities (4), (5) or (6) can be cast into the framework of the Stochastic Invariance Problem, by appropriately defining the sequence of sets Σk ⊂ X, k = 0, 1, ..., N .