Parallel Coordinate Descent Newton Method for Efficient $L_{1}$ -Regularized Loss Minimization

The recent years have witnessed advances in parallel algorithms for large-scale optimization problems. Notwithstanding the demonstrated success, existing algorithms that parallelize over features are usually limited by divergence issues under high parallelism or require data preprocessing to alleviate these problems. In this paper, we propose a Parallel Coordinate Descent algorithm using <italic>approximate</italic> Newton steps (PCDN) that is guaranteed to converge globally without data preprocessing. The key component of the PCDN algorithm is the high-dimensional line search, which guarantees the global convergence with high parallelism. The PCDN algorithm randomly partitions the feature set into <inline-formula> <tex-math notation="LaTeX">$b$ </tex-math></inline-formula> subsets/bundles of size <inline-formula> <tex-math notation="LaTeX">$P$ </tex-math></inline-formula>, and sequentially processes each bundle by first computing the descent directions for each feature in parallel and then conducting <inline-formula> <tex-math notation="LaTeX">$P$ </tex-math></inline-formula>-dimensional line search to compute the step size. We show that: 1) the PCDN algorithm is guaranteed to converge globally despite increasing parallelism and 2) the PCDN algorithm converges to the specified accuracy <inline-formula> <tex-math notation="LaTeX">$\epsilon $ </tex-math></inline-formula> within the limited iteration number of <inline-formula> <tex-math notation="LaTeX">$T_\epsilon $ </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">$T_\epsilon $ </tex-math></inline-formula> decreases with increasing parallelism. In addition, the data transfer and synchronization cost of the <inline-formula> <tex-math notation="LaTeX">$P$ </tex-math></inline-formula>-dimensional line search can be minimized by maintaining intermediate quantities. For concreteness, the proposed PCDN algorithm is applied to <inline-formula> <tex-math notation="LaTeX">$L_{1}$ </tex-math></inline-formula>-regularized logistic regression and <inline-formula> <tex-math notation="LaTeX">$L_{1}$ </tex-math></inline-formula>-regularized <inline-formula> <tex-math notation="LaTeX">$L_{2}$ </tex-math></inline-formula>-loss support vector machine problems. Experimental evaluations on seven benchmark data sets show that the PCDN algorithm exploits parallelism well and outperforms the state-of-the-art methods.