Convex Optimization with Random Pursuit

Optimization problems are ubiquitous in science and engineering. In this thesis, we study unconstrained black-box optimization problems that can only be accessed by an oracle that returns the function value at a query point. The theory of convex optimization problems is welldeveloped and such problems are typically solved with gradient-based methods. For non-convex problems, there is no unifying theoretical treatment and one has to rely on, typically gradient-free, search heuristics. Here, we analyze gradient-free optimization algorithms on convex functions. In the first part of this thesis, we study Random Pursuit algorithms. These are iterative search schemes, where each iteration consists of two steps: (i) the generation of a (random) search direction and (ii) performing a step along this direction. We present a general framework to study such algorithms and prove convergence on smooth convex and strongly convex functions. The convergence rates depend on a sufficient decrease condition that measures the quality of the generated steps. This condition is for instance met by schemes that use a line search to generate the steps. For line search algorithms, we extend the convergence analysis to functions that are not necessarily everywhere strongly convex, but only at the optimum. Line search algorithms do not need any problem specific parameterization as input and are invariant under strictly monotone transformations of the objective functions. They thus enjoy identical convergence behavior on a wider function class. We discuss several kinds of random search directions and provide estimates for the expected convergence rates. In the second part, we present three, at first sight seemingly unrelated, optimization algorithms that can be analyzed in the Random Pursuit framework. The examples comprise (i) solving linear systems with Kaczmarz’ method and (ii) Hessian learning with Leventhal and