A Fast Alternating Direction Method of Multipliers Algorithm for Big Data Applications

In recent years, with the explosive growth of the data, a wide range of data in Cyber-Physical-Social Systems (CPSS) are generated and collected as big data. Cloud computing have been widely-used as the supporting computation infrastructure, which makes big data analysis gaining much attention from IT industry and academia. Moreover, the data often are distributed and stored in different computation resources in many big data applications. Therefore, distributed computing and optimization has been developed for solving big data problems in cloud computing. time efficiency is the significant bottleneck in the performance of distributed optimization algorithms. In this paper, we propose a novel fast distributed algorithm via Alternating Direction Method of Multipliers with Adaptive Local Update (ADMM-ALU), that uses an efficient adaptive local update strategy to accelerate the speed of convergence by automatically determining the number of inner iterations of local update in each outer iteration (communication round). In particular, our method applies the optimality conditions and the magnitudes of residuals of ADMM to freely steer the trade-off between communication and local computation. Empirically, the performance of our method is tested on several benchmark datasets, and the experimental results show that compared to various versions of ADMM algorithms, our method converges faster, and could be a highly effective and efficient algorithm for practical use in big data applications.

[1]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[2]  Md Zakirul Alam Bhuiyan,et al.  Joint Optimization of Offloading Utility and Privacy for Edge Computing Enabled IoT , 2020, IEEE Internet of Things Journal.

[3]  Martin J. Wainwright,et al.  Communication-efficient algorithms for statistical optimization , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[4]  Jianguo Li,et al.  Large-scale Supervised Hierarchical Feature Learning for Face Recognition , 2014, ArXiv.

[5]  Taiji Suzuki,et al.  Stochastic Dual Coordinate Ascent with Alternating Direction Method of Multipliers , 2014, ICML.

[6]  Laurence T. Yang,et al.  A Tensor-Based Big-Data-Driven Routing Recommendation Approach for Heterogeneous Networks , 2019, IEEE Network.

[7]  Meikang Qiu,et al.  Feedback Dynamic Algorithms for Preemptable Job Scheduling in Cloud Systems , 2010 .

[8]  Dan Roth,et al.  Distributed Box-Constrained Quadratic Optimization for Dual Linear SVM , 2015, ICML.

[9]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[10]  Chia-Hua Ho,et al.  Recent Advances of Large-Scale Linear Classification , 2012, Proceedings of the IEEE.

[11]  B. He,et al.  Alternating Direction Method with Self-Adaptive Penalty Parameters for Monotone Variational Inequalities , 2000 .

[12]  Xuyun Zhang,et al.  An edge computing-enabled computation offloading method with privacy preservation for internet of connected vehicles , 2019, Future Gener. Comput. Syst..

[13]  Xiaokang Wang,et al.  NQA , 2019, ACM Trans. Embed. Comput. Syst..

[14]  Xuyun Zhang,et al.  A QoS-aware virtual machine scheduling method for energy conservation in cloud-based cyber-physical systems , 2019, World Wide Web.

[15]  Alexander J. Smola,et al.  Parallelized Stochastic Gradient Descent , 2010, NIPS.

[16]  Laurence T. Yang,et al.  A Distributed Tensor-Train Decomposition Method for Cyber-Physical-Social Services , 2019, ACM Trans. Cyber Phys. Syst..

[17]  Dan Roth,et al.  Distributed Training of Structured SVM , 2015, ArXiv.

[18]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[19]  Xuyun Zhang,et al.  Finding All You Need: Web APIs Recommendation in Web of Things Through Keywords Search , 2019, IEEE Transactions on Computational Social Systems.

[20]  Ohad Shamir,et al.  Optimal Distributed Online Prediction Using Mini-Batches , 2010, J. Mach. Learn. Res..

[21]  Alexander J. Smola,et al.  Communication Efficient Distributed Machine Learning with the Parameter Server , 2014, NIPS.

[22]  Yinghuan Shi,et al.  Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification , 2017, IEEE Transactions on Cybernetics.

[23]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[24]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[25]  Qiang He,et al.  An IoT-Oriented data placement method with privacy preservation in cloud environment , 2018, J. Netw. Comput. Appl..

[26]  Kang G. Shin,et al.  Efficient Distributed Linear Classification Algorithms via the Alternating Direction Method of Multipliers , 2012, AISTATS.

[27]  Michael I. Jordan,et al.  L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework , 2015, ArXiv.

[28]  Lianyong Qi,et al.  Privacy-Aware Multidimensional Mobile Service Quality Prediction and Recommendation in Distributed Fog Environment , 2018, Wirel. Commun. Mob. Comput..

[29]  Tianbao Yang,et al.  Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent , 2013, NIPS.

[30]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[31]  Jin Sun,et al.  Improving Availability of Multicore Real-Time Systems Suffering Both Permanent and Transient Faults , 2019, IEEE Transactions on Computers.

[32]  Zheng Xu,et al.  Adaptive ADMM with Spectral Penalty Parameter Selection , 2016, AISTATS.

[33]  Leon Wenliang Zhong,et al.  Fast Stochastic Alternating Direction Method of Multipliers , 2013, ICML.

[34]  Thomas Hofmann,et al.  Communication-Efficient Distributed Dual Coordinate Ascent , 2014, NIPS.

[35]  Xuyun Zhang,et al.  A computation offloading method over big data for IoT-enabled cloud-edge computing , 2019, Future Gener. Comput. Syst..

[36]  Min Li,et al.  Adaptive Primal-Dual Splitting Methods for Statistical Learning and Image Processing , 2015, NIPS.

[37]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[38]  Konstantina Christakopoulou,et al.  Accelerated Alternating Direction Method of Multipliers , 2015, KDD.

[39]  Chao Yan,et al.  Link prediction in paper citation network to construct paper correlation graph , 2019, EURASIP J. Wirel. Commun. Netw..

[40]  Chih-Jen Lin,et al.  A dual coordinate descent method for large-scale linear SVM , 2008, ICML '08.

[41]  Meikang Qiu,et al.  Energy minimization with loop fusion and multi-functional-unit scheduling for multidimensional DSP , 2008, J. Parallel Distributed Comput..

[42]  Vladimir Pavlovic,et al.  Fast ADMM Algorithm for Distributed Optimization with Adaptive Penalty , 2015, AAAI.

[43]  Junlong Zhou,et al.  Security-Critical Energy-Aware Task Scheduling for Heterogeneous Real-Time MPSoCs in IoT , 2020, IEEE Transactions on Services Computing.

[44]  Meikang Qiu,et al.  Keystroke Biometric Systems for User Authentication , 2017, J. Signal Process. Syst..

[45]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[46]  Tao Huang,et al.  An energy-aware computation offloading method for smart edge computing in wireless metropolitan area networks , 2019, J. Netw. Comput. Appl..

[47]  Laurence T. Yang,et al.  A Cloud-Edge Computing Framework for Cyber-Physical-Social Services , 2017, IEEE Communications Magazine.

[48]  João M. F. Xavier,et al.  D-ADMM: A Communication-Efficient Distributed Algorithm for Separable Optimization , 2012, IEEE Transactions on Signal Processing.