A Bregman extension of quasi-Newton updates II: Analysis of robustness properties

In Part I of this series of articles, we introduced the information geometric framework of quasi-Newton methods and gave an extension of Hessian update formulas based on the Bregman divergence. The purpose of this article is to investigate the convergence and robustness properties of extended Hessian update formulas. Fletcher has studied a variational problem which derives the approximate Hessian update formula of the quasi-Newton methods. We point out that the variational problem is identical to optimization of the Kullback-Leibler divergence, which is a discrepancy measure between two probability distributions. Then, we introduce the Bregman divergence as an extension of the Kullback-Leibler divergence, and derive extended quasi-Newton update formulas based on the variational problem with the Bregman divergence. The proposed update formulas belong to a class of self-scaling quasi-Newton methods. We study the convergence property of the proposed quasi-Newton method. Moreover, we apply tools in the robust statistics to analyze the robustness properties of Hessian update formulas against numerical rounding errors or a shift of tuning parameters included in line search methods of the step length. As the main contribution of this paper, we present that the influence of perturbations in the line search is bounded only for the standard BFGS formula for the Hessian approximation. Numerical studies are conducted to verify the usefulness of the tools borrowed from the robust statistics.