Variance Reduction via Primal-Dual Accelerated Dual Averaging for Nonsmooth Convex Finite-Sums

Structured nonsmooth convex finite-sum optimization appears in many machine learning applications, including support vector machines and least absolute deviation. For the primaldual formulation of this problem, we propose a novel algorithm called Variance Reduction via Primal-Dual Accelerated Dual Averaging (VRPDA2). In the nonsmooth and general convex setting, VRPDA2 has the overall complexity O(nd logmin{1/✏, n}+ d/✏) in terms of the primal-dual gap, where n denotes the number of samples, d the dimension of the primal variables, and ✏ the desired accuracy. In the nonsmooth and strongly convex setting, the overall complexity of VRPDA2 becomes O(nd logmin{1/✏, n}+d/ p ✏) in terms of both the primal-dual gap and the distance between iterate and optimal solution. Both these results for VRPDA2 improve significantly on state-of-the-art complexity estimates— which are O(nd logmin{1/✏, n} + p nd/✏) for the nonsmooth and general convex setting and O(nd logmin{1/✏, n}+ p nd/ p ✏) for the nonsmooth and strongly convex setting—with a simpler and more straightforward algorithm and analysis. Moreover, both complexities are better than lower bounds for general convex finite-sum optimization, because our approach makes use of additional, commonly occurring structure. Numerical experiments reveal competitive performance of VRPDA2 compared to state-of-the-art approaches.