Calculating Principal Eigen-Functions of Non-Negative Integral Kernels: Particle Approximations and Applications

Often in applications such as rare events estimation or optimal control it is required that one calculates the principal eigenfunction and eigenvalue of a nonnegative integral kernel. Except in the finite-dimensional case, usually neither the principal eigenfunction nor the eigenvalue can be computed exactly. In this paper, we develop numerical approximations for these quantities. We show how a generic interacting particle algorithm can be used to deliver numerical approximations of the eigenquantities and the associated so-called “twisted” Markov kernel as well as how these approximations are relevant to the aforementioned applications. In addition, we study a collection of random integral operators underlying the algorithm, address some of their mean and pathwise properties, and obtain error estimates. Finally, numerical examples are provided in the context of importance sampling for computing tail probabilities of Markov chains and computing value functions for a class of stochastic optimal control pro...

[1]  A. Doucet,et al.  Particle Motions in Absorbing Medium with Hard and Soft Obstacles , 2004 .

[2]  Gersende Fort,et al.  Forgetting the initial distribution for Hidden Markov Models , 2007 .

[3]  P. Ney,et al.  Monte Carlo simulation and large deviations theory for uniformly recurrent Markov chains , 1990, Journal of Applied Probability.

[4]  P. Dupuis,et al.  Dynamic importance sampling for uniformly recurrent Markov chains , 2005, math/0503454.

[5]  Sumeetpal S. Singh,et al.  A backward particle interpretation of Feynman-Kac formulae , 2009, 0908.2556.

[6]  P. Ney,et al.  Large deviations of uniformly recurrent Markov additive processes , 1985 .

[7]  Wendell H. Fleming,et al.  Advances in Filtering and Optimal Stochastic Control , 1982 .

[8]  K. Athreya.  Change of measures for Markov chains and the LlogL theorem for branching processes , 2000 .

[9]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[10]  Sean P. Meyn,et al.  Risk-Sensitive Optimal Control for Markov Decision Processes with Monotone Cost , 2002, Math. Oper. Res..

[11]  Wolfgang J. Runggaldier,et al.  Connections between stochastic control and dynamic games , 1996, Math. Control. Signals Syst..

[12]  P. Ney.  GENERAL IRREDUCIBLE MARKOV CHAINS AND NON‐NEGATIVE OPERATORS (Cambridge Tracts in Mathematics, 83) , 1986 .

[13]  A. Veretennikov,et al.  On discrete time ergodic filters with wrong initial data , 2008 .

[14]  N. Kantas.  Sequential decision making in general state space models , 2009 .

[15]  T. E. Harris,et al.  The Theory of Branching Processes. , 1963 .

[16]  Pierre Del Moral,et al.  Mean Field Simulation for Monte Carlo Integration , 2013 .

[17]  Pierre Del Moral,et al.  On the Robustness of the Snell Envelope , 2011, SIAM J. Financial Math..

[18]  Mathias Rousset,et al.  On the Control of an Interacting Particle Estimation of Schrödinger Ground States , 2006, SIAM J. Math. Anal..

[19]  Sanjoy K. Mitter,et al.  A Variational Approach to Nonlinear Estimation , 2003, SIAM J. Control. Optim..

[20]  F. Albertini,et al.  Logarithmic transformations for discrete-time, finite-horizon stochastic control problems , 1988 .

[21]  P. Del Moral,et al.  Snell Envelope with Small Probability Criteria , 2012, Applied Mathematics & Optimization.

[22]  T. Alderweireld,et al.  A Theory for the Term Structure of Interest Rates , 2004, cond-mat/0405293.

[23]  S. Mitter,et al.  Optimal control and nonlinear filtering for nondegenerate diffusion processes , 1982 .

[24]  Caffarel,et al.  Diffusion monte carlo methods with a fixed number of walkers , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[25]  R. Laubenfels,et al.  Feynman–Kac Formulae: Genealogical and Interacting Particle Systems with Applications , 2005 .

[26]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[27]  Aurélien Garivier,et al.  Sequential Monte Carlo smoothing for general state space hidden Markov models , 2011, 1202.2945.

[28]  E. Nummelin.  General irreducible Markov chains and non-negative operators: List of symbols and notation , 1984 .

[29]  Benjamin Jourdain,et al.  Diffusion Monte Carlo method: numerical analysis in a simple case , 2007 .

[30]  S. Sheu.  Stochastic control and principal eigenvaluet , 1984 .

[31]  W. Fleming.  Logarithmic Transformations and Stochastic Control , 1982 .

[32]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[33]  Hilbert J. Kappen,et al.  Online solution of the average cost kullback-leibler optimization problem. , 2011 .

[34]  L. Rogers,et al.  Diffusions, Markov Processes and Martingales - Vol 1: Foundations , 1979 .

[35]  N. Whiteley.  Stability properties of some particle filters , 2011, 1109.6779.

[36]  Nick Whiteley,et al.  Linear Variance Bounds for Particle Approximations of Time-Homogeneous Feynman-Kac Formulae , 2012 .

[37]  Tze Leung Lai,et al.  A sequential Monte Carlo approach to computing tail probabilities in stochastic models , 2011 .

[38]  O. Hernández-Lerma,et al.  Discrete-time Markov control processes , 1999 .

[39]  E. Todorov,et al.  A UNIFIED THEORY OF LINEARLY SOLVABLE OPTIMAL CONTROL , 2012 .

[40]  Pierre Del Moral,et al.  Particle approximations of Lyapunov exponents connected to Schrödinger operators and Feynman–Kac semigroups , 2003 .

[41]  P. Ney,et al.  Markov Additive Processes I. Eigenvalue Properties and Limit Theorems , 1987 .

[42]  Emanuel Todorov,et al.  General duality between optimal control and estimation , 2008, 2008 47th IEEE Conference on Decision and Control.

[43]  Stefan Schaal,et al.  A Generalized Path Integral Control Approach to Reinforcement Learning , 2010, J. Mach. Learn. Res..

[44]  S. Meyn,et al.  Spectral theory and limit theorems for geometrically ergodic Markov processes , 2002, math/0209200.

[45]  Peter March,et al.  A Fleming–Viot Particle Representation¶of the Dirichlet Laplacian , 2000 .

[46]  H. Kappen.  Linear theory for control of nonlinear stochastic systems. , 2004, Physical review letters.

[47]  P. Collet,et al.  Quasi-Stationary Distributions: Markov Chains, Diffusions and Dynamical Systems , 2012 .