Proper Policies in Infinite-State Stochastic Shortest Path Problems

We consider stochastic shortest path problems with infinite state and control spaces, a nonnegative cost per stage, and a termination state. We extend the notion of a proper policy, a policy that terminates within a finite expected number of steps, from the context of finite state space to the context of infinite state space. We consider the optimal cost function <inline-formula><tex-math notation="LaTeX">$J^*$</tex-math></inline-formula>, and the optimal cost function <inline-formula><tex-math notation="LaTeX">$\hat{J}$</tex-math></inline-formula> over just the proper policies. We show that <inline-formula><tex-math notation="LaTeX">$J^*$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$\hat{J}$</tex-math></inline-formula> are the smallest and largest solutions of Bellman's equation, respectively, within a suitable class of Lyapounov-like functions. If the cost per stage is bounded, these functions are those that are bounded over the effective domain of <inline-formula> <tex-math notation="LaTeX">$\hat{J}$</tex-math></inline-formula>. The standard value iteration algorithm may be attracted to either <inline-formula><tex-math notation="LaTeX">$J^*$</tex-math></inline-formula> or <inline-formula> <tex-math notation="LaTeX">$\hat{J}$</tex-math></inline-formula>, depending on the initial condition.