Resetting in stochastic optimal control

``When in a difficult situation, it is sometimes better to give up and start all over again''. While this empirical truth has been regularly observed in a wide range of circumstances, quantifying the effectiveness of such a heuristic strategy remains an open challenge. In this paper, we combine the notions of optimal control and stochastic resetting to address this problem. The emerging analytical framework allows not only to measure the performance of a given restarting policy but also to obtain the optimal strategy for a wide class of dynamical systems. We apply our technique to a system with a final reward and show that the reward value must be larger than a critical threshold for resetting to be effective. Our approach, analogous to the celebrated Hamilton-Jacobi-Bellman paradigm, provides the basis for the investigation of realistic restarting strategies across disciplines. As an application, we show that the framework can be applied to an epidemic model to predict the optimal lockdown policy.

[1]  S. Majumdar,et al.  Distribution of the time of the maximum for stationary processes , 2021, EPL (Europhysics Letters).

[2]  S. Majumdar,et al.  Dynamical phase transition in the first-passage probability of a Brownian motion. , 2021, Physical review. E.

[3]  S. Sinha,et al.  Optimal control in pandemics. , 2021, Physical review. E.

[4]  U. Boscain,et al.  Introduction to the Pontryagin Maximum Principle for Quantum Optimal Control , 2020, PRX Quantum.

[5]  S. Redner,et al.  Optimization and growth in first-passage resetting , 2020, 2009.03419.

[6]  S. Redner,et al.  Optimization in First-Passage Resetting. , 2020, Physical review letters.

[7]  S. Majumdar,et al.  Optimal mean first-passage time for a Brownian searcher subjected to resetting: Experimental and theoretical results , 2020, Physical Review Research.

[8]  Arnab K. Pal,et al.  Experimental Realization of Diffusion with Stochastic Resetting , 2020, The journal of physical chemistry letters.

[9]  S. Majumdar,et al.  Stochastic resetting and applications , 2019, Journal of Physics A: Mathematical and Theoretical.

[10]  Alexandre Dolgui,et al.  Scheduling in production, supply chain and Industry 4.0 systems by optimal control: fundamentals, state-of-the-art and applications , 2019, Int. J. Prod. Res..

[11]  Vicenç Méndez,et al.  Transport properties and first-arrival statistics of random motion with stochastic reset times. , 2018, Physical review. E.

[12]  Stephan Eule,et al.  Non-equilibrium steady states of stochastic processes with intermittent resetting , 2015, 1510.07876.

[13]  Vicenç Méndez,et al.  Phase transitions in optimal search times: How random walkers should combine resetting and flight scales. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[14]  S. Majumdar,et al.  Effective Langevin equations for constrained stochastic processes , 2015, 1503.02639.

[15]  Satya N Majumdar,et al.  Dynamical transition in the temporal relaxation of stochastic processes under resetting. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  Vicenç Gómez,et al.  Real-Time Stochastic Optimal Control for Multi-Agent Quadrotor Systems , 2015, ICAPS.

[17]  Lukasz Kusmierz,et al.  First Order Transition for the Optimal Search Time of Lévy Flights with Resetting. , 2014, Physical review letters.

[18]  C. Landim,et al.  Macroscopic fluctuation theory , 2014, 1404.6466.

[19]  Shlomi Reuveni,et al.  Role of substrate unbinding in Michaelis–Menten enzymatic reactions , 2014, Proceedings of the National Academy of Sciences.

[20]  M. Srinivasan.  Search for Research , 2014, Journal of conservative dentistry : JCD.

[21]  Satya N. Majumdar,et al.  Diffusion with optimal resetting , 2011, 1107.4225.

[22]  Satya N Majumdar,et al.  Diffusion with stochastic resetting. , 2011, Physical review letters.

[23]  A. Sobolevski,et al.  Ballistic deposition patterns beneath a growing Kardar-Parisi-Zhang interface. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[24]  S. Majumdar,et al.  Switching and growth for microbial populations in catastrophic responsive environments. , 2009, Biophysical journal.

[25]  Huyen Pham,et al.  Continuous-time stochastic control and optimization with financial applications / Huyen Pham , 2009 .

[26]  U. Seifert,et al.  Optimal finite-time processes in stochastic thermodynamics. , 2007, Physical review letters.

[27]  M. Shlesinger.  Mathematical physics: Search research , 2006, Nature.

[28]  H. Kappen.  Linear theory for control of nonlinear stochastic systems. , 2004, Physical review letters.

[29]  Antonis Economou,et al.  A continuous-time Markov chain under the influence of a regulating point process and applications in stochastic models with catastrophes , 2003, Eur. J. Oper. Res..

[30]  Andrea Montanari,et al.  Optimizing searches via rare events. , 2002, Physical review letters.

[31]  S. Majumdar,et al.  Persistence in nonequilibrium systems , 1999, cond-mat/9907407.

[32]  M. James.  Controlled markov processes and viscosity solutions , 1994 .

[33]  W. Fleming,et al.  Controlled Markov processes and viscosity solutions , 1992 .

[34]  Robert E. Kalaba,et al.  Dynamic Programming and Modern Control Theory , 1966 .

[35]  M. L. Chambers.  The Mathematical Theory of Optimal Processes , 1965 .

[36]  W. O. Kermack,et al.  A contribution to the mathematical theory of epidemics , 1927 .

[37]  Benjamin Pfaff,et al.  Free And Moving Boundary Problems , 2016 .

[38]  A. Sobolevski,et al.  Ju l 2 01 0 Ballistic deposition patterns beneath a growing KPZ interface , 2010 .

[39]  P. Hippe,et al.  Optimal Control and Estimation , 2009 .