Perseus: A Simple High-Order Regularization Method for Variational Inequalities

This paper settles an open and challenging question pertaining to the design of simple high-order regularization methods for solving smooth and monotone variational inequalities (VIs). A VI involves finding x ⋆ ∈ X such that h F ( x ) , x − x ⋆ i ≥ 0 for all x ∈ X and we consider the setting where F : R d 7→ R d is smooth with up to ( p − 1) th -order derivatives. For the case of p = 2, Nesterov [2006] extended the cubic regularized Newton’s method to VIs with a global rate of O ( ǫ − 1 ). Monteiro and Svaiter [2012] proposed another second-order method which achieved an improved rate of O ( ǫ − 2 / 3 log(1 /ǫ )), but this method required a nontrivial binary search procedure as an inner loop. High-order methods based on similar binary search procedures have been further developed and shown to achieve a rate of O ( ǫ − 2 / ( p +1) log(1 /ǫ )) [Bullins and Lai, 2020, Lin and Jordan, 2021b, Jiang and Mokhtari, 2022]. However, such search procedure can be computationally prohibitive in practice [Nesterov, 2018] and the problem of finding a simple high-order regularization methods remains as an open and challenging question in optimization theory. We propose a p th -order method which does not require any binary search scheme and is guaranteed to converge to a weak solution with a global rate of O ( ǫ − 2 / ( p +1) ). A version with restarting attains a global linear and local superlinear convergence rate for smooth and strongly monotone VIs. Further, our method achieves a global rate of O ( ǫ − 2 /p ) for solving smooth and non-monotone VIs satisfying the Minty condition; moreover, the restarted version again attains a global linear and local superlinear convergence rate if the strong Minty condition holds.