A tutorial on event-based optimization—a new optimization framework

In many practical systems, the control or decision making is triggered by certain events. The performance optimization of such systems is generally different from the traditional optimization approaches, such as Markov decision processes or dynamic programming. The goal of this tutorial is to introduce, in an intuitive manner, a new optimization framework called event-based optimization. This framework has a wide applicability to aforementioned systems. With performance potential as building blocks, we develop two intuitive optimization algorithms to solve the event-based optimization problem. The optimization algorithms are proposed based on an intuitive principle, and theoretical justifications are given with a performance sensitivity based approach. Finally, we provide a few practical examples to demonstrate the effectiveness of the event-based optimization framework. We hope this framework may provide a new perspective to the optimization of the performance of event-triggered dynamic systems.

[1]  Xi-Ren Cao,et al.  Perturbation realization, potentials, and sensitivity analysis of Markov processes , 1997, IEEE Trans. Autom. Control..

[2]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[3]  Ward Whitt,et al.  Approximations of Dynamic Programs, I , 1978, Math. Oper. Res..

[4]  Li Xia Optimal control of customer admission to an open Jackson network , 2012, Proceedings of the 31st Chinese Control Conference.

[5]  Ronald E. Parr,et al.  Hierarchical control and learning for markov decision processes , 1998 .

[6]  Qing-Shan Jia,et al.  On Solving Event-Based Optimization With Average Reward Over Infinite Stages , 2011, IEEE Transactions on Automatic Control.

[7]  Xi-Ren Cao,et al.  Stochastic Learning and Optimization: A Sensitivity-Based Approach (International Series on Discrete Event Dynamic Systems) , 2007 .

[8]  E. Altman Constrained Markov Decision Processes , 1999 .

[9]  Zhiyuan Ren,et al.  A time aggregation approach to Markov decision processes , 2002, Autom..

[10]  Li Xia,et al.  Event-based optimization of admission control in open queueing networks , 2014, Discret. Event Dyn. Syst..

[11]  Arie Hordijk,et al.  Perturbation analysis of waiting times in the G/G/1 queue , 2013, Discret. Event Dyn. Syst..

[12]  Xi-Ren Cao,et al.  Realization Probabilities: The Dynamics of Queuing Systems , 1994 .

[13]  Xi-Ren Cao,et al.  Basic Ideas for Event-Based Optimization of Markov Systems , 2005, Discret. Event Dyn. Syst..

[14]  Chao-Bo Yan,et al.  Efficient Simulation Method for General Assembly Systems With Material Handling Based on Aggregated Event-Scheduling , 2010, IEEE Transactions on Automation Science and Engineering.

[15]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[16]  Christos G. Cassandras,et al.  Using infinitesimal perturbation analysis of stochastic flow models to recover performance sensitivity estimates of discrete event systems , 2012, Discret. Event Dyn. Syst..

[17]  Xi-Ren Cao,et al.  Event-Based Optimization for POMDPs and Its Application in Portfolio Management , 2011 .

[18]  John N. Tsitsiklis,et al.  Simulation-based optimization of Markov reward processes , 2001, IEEE Trans. Autom. Control..

[19]  Xi-Ren Cao,et al.  Stochastic learning and optimization - A sensitivity-based approach , 2007, Annu. Rev. Control..

[20]  Xi-Ren Cao,et al.  Event-Based Optimization of Markov Systems , 2008, IEEE Transactions on Automatic Control.

[21]  Sridhar Mahadevan,et al.  Recent Advances in Hierarchical Reinforcement Learning , 2003, Discret. Event Dyn. Syst..

[22]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[23]  Javier de Frutos,et al.  Approximation of Dynamic Programs , 2012 .

[24]  Li Xia,et al.  Performance optimization of queueing systems with perturbation realization , 2012, Eur. J. Oper. Res..

[25]  Christos G. Cassandras,et al.  Perturbation Analysis of Discrete Event Systems , 2015, Encyclopedia of Systems and Control.

[26]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[27]  E. Altman,et al.  Perturbation analysis for denumerable Markov chains with application to queueing models , 2004, Advances in Applied Probability.