Implementing the ADMM to Big Datasets : A Case Study of LASSO

The alternating direction method of multipliers (ADMM) has been popularly used for a wide range of applications in the literature. When big datasets with high-dimensional variables are considered, subproblems arising from the ADMM must be solved inexactly even though theoretically they may have closed-form solutions. Such a scenario immediately poses mathematical ambiguities such as how accurately these subproblems should be solved and whether or not the convergence can be still guaranteed. Despite of the popularity of ADMM, it seems not too much is known in these regards. In this paper, we look into the mathematical detail of implementing the ADMM to such big-data scenarios. More specifically, we focus on the convex programming case where there is a quadratic function component with extremely high-dimensional variables in the objective of the model under discussion and thus there is a huge-scale system of linear equations to be solved at each iteration of the ADMM. We show that there is no need (indeed it is impossible) to solve this linear system exactly or too accurately; and propose an automatically adjustable inexactness criterion to solve these linear systems inexactly. We further identify the safe-guard numbers for the internally nested iterations that can sufficiently ensure this inexactness criterion if these linear systems are solved by standard numerical linear algebra solvers. The convergence, together with worst-case convergence rate measured by the iteration complexity, is rigorously established for the ADMM with inexactly-solved subproblems. Some numerical experiments for big datasets of the LASSO with millions of variables are reported to show the efficiency of this inexact implementation of ADMM.