On Solving Event-Based Optimization With Average Reward Over Infinite Stages

Event-based optimization (EBO) provides a unified framework for problems in which decisions can be made only when certain events occur. Because the event sequence usually is not Markovian, the optimal policy could depend on the entire event history, which is hard to implement in practice. So most existing studies focus on memoryless policies, which make decisions only based on the current observable events. But it remains open how to find the optimal memoryless policies in general, leaving alone to solve the EBO optimally. In this technical note, we address these two important questions for infinite-stage EBOs with finite state and action spaces and make the following three major contributions. First, we extend our previous studies on finite-stage EBOs and convert infinite-stage EBOs to partially observable Markov decision processes (POMDPs). The belief process of this POMDP is called belief-event decision process (BEDP). Under certain well-known conditions, the optimal policies of BEDPs can be achieved within stationary Markov deterministic policies. Second, assuming optimal stationary policies exist, the performance difference and derivative formulas are developed. Potentials of memoryless event-based policies are shown to be piecewise linear functions, and thus can be efficiently estimated through sample paths. Third, a potential-based approximate policy iteration algorithm is developed to obtain near-optimal memoryless policies. The convergence and performance loss bound of this algorithm are analyzed.