Error Bound Analysis of $Q$ -Function for Discounted Optimal Control Problems With Policy Iteration

In this paper, we present error bound analysis of the <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-function for the action-dependent adaptive dynamic programming for solving discounted optimal control problems of unknown discrete-time nonlinear systems. The convergence of <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-functions derived by a policy iteration algorithm under ideal conditions is given. Considering the approximated errors of the <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-function and control policy in the policy evaluation step and policy improvement step, we establish error bounds of approximate <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-functions in each iteration. With the given boundedness conditions, the approximate <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-function will converge to a finite neighborhood of the optimal <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-function. To implement the presented algorithm, two three-layer neural networks are employed to approximate the <inline-formula> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula>-function and the control policy, respectively. Finally, a simulation example is utilized to verify the validity of the presented algorithm.