POLO: a POLicy-based Optimization library

We present POLO --- a C++ library for large-scale parallel optimization research that emphasizes ease-of-use, flexibility and efficiency in algorithm design. It uses multiple inheritance and template programming to decompose algorithms into essential policies and facilitate code reuse. With its clear separation between algorithm and execution policies, it provides researchers with a simple and powerful platform for prototyping ideas, evaluating them on different parallel computing architectures and hardware platforms, and generating compact and efficient production code. A C-API is included for customization and data loading in high-level languages. POLO enables users to move seamlessly from serial to multi-threaded shared-memory and multi-node distributed-memory executors. We demonstrate how POLO allows users to implement state-of-the-art asynchronous parallel optimization algorithms in just a few lines of code and report experiment results from shared and distributed-memory computing architectures. We provide both POLO and POLO.jl, a wrapper around POLO written in the Julia language, at this https URL under the permissive MIT license.

[1]  Sanjiv Kumar,et al.  On the Convergence of Adam and Beyond , 2018 .

[2]  Rishabh K. Iyer,et al.  Jensen: An Easily-Extensible C++ Toolkit for Production-Level Machine Learning and Convex Optimization , 2018, ArXiv.

[3]  Jared Hoberock,et al.  Parallel Algorithms Need Executors | N4406 , 2015 .

[4]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[5]  J. Koenderink.  Q… , 2014, Les noms officiels des communes de Wallonie, de Bruxelles-Capitale et de la communaute germanophone.

[6]  Alexander J. Smola,et al.  AdaDelay: Delay Adaptive Distributed Stochastic Convex Optimization , 2015, ArXiv.

[7]  Michael I. Jordan,et al.  Ray: A Distributed Framework for Emerging AI Applications , 2017, OSDI.

[8]  Alfred O. Hero,et al.  A Convergent Incremental Gradient Method with a Constant Step Size , 2007, SIAM J. Optim..

[10]  Aaron Q. Li,et al.  Parameter Server for Distributed Machine Learning , 2013 .

[11]  Conrad Sanderson,et al.  Armadillo: a template-based C++ library for linear algebra , 2016, J. Open Source Softw..

[12]  Michael Minkoff,et al.  Randomly Generated Test Problems for Positive Definite Quadratic Programming , 1984, TOMS.

[13]  Hamid Reza Feyzmahdavian,et al.  Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Server , 2016, ArXiv.

[14]  Fabian Pedregosa,et al.  Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization , 2017, NIPS.

[15]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[16]  Francis Bach,et al.  SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives , 2014, NIPS.

[17]  Weizhu Chen,et al.  DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization , 2017, J. Mach. Learn. Res..

[18]  Andrei Alexandrescu,et al.  Modern C++ Design , 2001 .

[19]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[20]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[21]  Albert Cohen,et al.  Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions , 2018, ArXiv.

[22]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[25]  Timothy Dozat,et al.  Incorporating Nesterov Momentum into Adam , 2016 .

[26]  Fabian Pedregosa,et al.  ASAGA: Asynchronous Parallel SAGA , 2016, AISTATS.

[27]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .