SVRG + + with Non-uniform Sampling

SVRG++ is a recent randomized optimization algorithm designed to solve nonstrongly convex smooth composite optimization problems in the large data regime. In this paper we combine SVRG++ with non-uniform sampling of the data points (already present in the original SVRG algorithm), leading to an algorithm with the best sample complexity to date and state-of-the art empirical performance. While the combination and the analysis of the algorithm is admittedly straightforward, our experimental results show significant improvement over the original SVRG++ method with the new method outperforming all competitors on datasets where the smoothness of the components varies. This demonstrates that, despite its simplicity and limited novelty, this extension is important in practice.