Parallel Support Vector Machines on Multi-Core and Multiprocessor Systems

This paper proposes a new and efficient parallel implementation of support vector machines based on decomposition method for handling large scale datasets. The parallelizing is performed on the most time-and-memory consuming work of training, i.e., to update the vector f . The inner problems are dealt by sequential minimal optimization solver. Since the underlying parallelism is realized by the shared memory version of Map-Reduce paradigm, our system is easy to build and particularly suitable to apply to multi-core and multiprocessor systems. Experimental results show that on most of the tested datasets, our system offers higher than fou rfold increase in speed compared to Libsvm, and it is also far more efficient than the MPI implementation Pisvm. Keywords-support vector machine; parallel; multi-core; Map-Reduce I. I NTRODUCTION Support vector machine (SVM) is a popular supervised learning method of solving classification and regression problems [1]. It shows robust generalization ability in numerous applications such as image processing, text mining, neural analysis and energy efficiency modeling [2], [3]. The training of SVM is essentially a quadratic optimization pro blem which is both time and memory costly while running on computers, making it a challenge to apply SVM on large scale problems. Several optimizing or heuristic methods have been proposed to accelerate the training and reduce the memory occupation, such as shrinking, chunking [4], kernel caching [5], approximation of kernel matrix [6]. In additio n, certain scalable solvers can be used such as Sequential Minimal Optimization (SMO) [7], mixture SVMs [8], primal estimated sub-gradient solver [9]. Despite these efforts, however, a more sophisticated and satisfactory solution is still expected for this challenging research problem. Thanks to the modern chip manufacturing, we are entering the multi-core era. Computers with multi-cores or multipro cessors are becoming more available and affordable. This paper aims to investigate and demonstrate how SVM — a popular machine learning algorithm can benefit from this modern platform. A new parallel SVM that is particularly suitable to shared memory system is proposed. Decomposition method, caching and SMO inner quadratic problem solver are composed in the implementation as the key techniques. For the purpose of achieving easy implementation without sacrificing performance, the state-of-the-art par allel programming framework Map-Reduce is chosen to perform the underlying parallelism. The system is therefore called MRPsvm, which stands for ”Map-Reduce parallel SVM”. Comparative system analysis and experimental results on benchmark datasets show significant memory saving and overwhelming speed increase in our system. The following sections are organized as follows. Section II introduces the basic problem of SVM training and how the decomposition method can solve the problem. Section III explains the some key points of system implementation. Section IV describes the related work. Section V presents the numerical experiments on benchmark datasets and corresponding results. Conclusions are drawn in section VI. II. SUPPORT VECTOR MACHINE SVM for classification purpose aims at finding a hyperplane to separate two classes with maximum margin. Let xi denotes theith training sample,yi denotes the corresponding label with the value either -1 or 1, i = 1, 2, ..., l. l is the total number of training samples. The dual form of the SVM can be written as the following convex quadratic function.