A distributed semi-supervised learning algorithm based on manifold regularization using wavelet neural network

This paper aims to propose a distributed semi-supervised learning (D-SSL) algorithm to solve D-SSL problems, where training samples are often extremely large-scale and located on distributed nodes over communication networks. Training data of each node consists of labeled and unlabeled samples whose output values or labels are unknown. These nodes communicate in a distributed way, where each node has only access to its own data and can only exchange local information with its neighboring nodes. In some scenarios, these distributed data cannot be processed centrally. As a result, D-SSL problems cannot be centrally solved by using traditional semi-supervised learning (SSL) algorithms. The state-of-the-art D-SSL algorithm, denoted as Distributed Laplacian Regularization Least Square (D-LapRLS), is a kernel based algorithm. It is essential for the D-LapRLS algorithm to estimate the global Euclidian Distance Matrix (EDM) with respect to total samples, which is time-consuming especially when the scale of training data is large. In order to solve D-SSL problems and overcome the common drawback of kernel based D-SSL algorithms, we propose a novel Manifold Regularization (MR) based D-SSL algorithm using Wavelet Neural Network (WNN) and Zero-Gradient-Sum (ZGS) distributed optimization strategy. Accordingly, each node is assigned an individual WNN with the same basis functions. In order to initialize the proposed D-SSL algorithm, we propose a centralized MR based SSL algorithm using WNN. We denote the proposed SSL and D-SSL algorithms as Laplacian WNN (LapWNN) and distributed LapWNN (D-LapWNN), respectively. The D-LapWNN algorithm works in a fully distributed fashion by using ZGS strategy, whose convergence is guaranteed by the Lyapunov method. During the learning process, each node only exchanges local coefficients with its neighbors rather than raw data. It means that the D-LapWNN algorithm is a privacy preserving method. At last, several illustrative simulations are presented to show the efficiency and advantage of the proposed algorithm.

[1]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[2]  Danilo Comminiello,et al.  A semi-supervised random vector functional-link network based on the transductive framework , 2016, Inf. Sci..

[3]  Vivek S. Borkar,et al.  Distributed and Asynchronous Methods for Semi-supervised Learning , 2016, WAW.

[4]  Xin Du,et al.  Distributed Semi-Supervised Metric Learning , 2016, IEEE Access.

[5]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[6]  Xiangyu Chang,et al.  Distributed Semi-supervised Learning with Kernel Ridge Regression , 2017, J. Mach. Learn. Res..

[7]  Weisheng Chen,et al.  Distributed learning for feedforward neural networks with random weights using an event-triggered communication scheme , 2017, Neurocomputing.

[8]  Zhiping Lin,et al.  Extreme Learning Machine for Joint Embedding and Clustering , 2018, Neurocomputing.

[9]  Weisheng Chen,et al.  A zero-gradient-sum algorithm for distributed cooperative learning using a feedforward neural network with random weights , 2016, Inf. Sci..

[10]  Kai Zhang,et al.  Extreme learning machine and adaptive sparse representation for image classification , 2016, Neural Networks.

[11]  Alexander Zien,et al.  Semi-Supervised Learning , 2006 .

[12]  Michelangelo Ceci,et al.  Semi-supervised trees for multi-target regression , 2018, Inf. Sci..

[13]  Weisheng Chen,et al.  A general framework for population-based distributed optimization over networks , 2017, Inf. Sci..

[14]  Nan Chen,et al.  Constrained NMF-based semi-supervised learning for social media spammer detection , 2017, Knowl. Based Syst..

[15]  Günther Palm,et al.  Semi-supervised learning for tree-structured ensembles of RBF networks with Co-Training , 2010, Neural Networks.

[16]  Zhiping Lin,et al.  Composite function wavelet neural networks with extreme learning machine , 2010, Neurocomputing.

[17]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[18]  Guang-Bin Huang,et al.  Trends in extreme learning machines: A review , 2015, Neural Networks.

[19]  Jun'ichi Takeuchi,et al.  Safe semi-supervised learning based on weighted likelihood , 2014, Neural Networks.

[20]  Mikhail Belkin,et al.  Laplacian Support Vector Machines Trained in the Primal , 2009, J. Mach. Learn. Res..

[21]  Choon Yik Tang,et al.  Zero-gradient-sum algorithms for distributed convex optimization: The continuous-time case , 2011, Proceedings of the 2011 American Control Conference.

[22]  Yuan Yan Tang,et al.  Convergence rate of the semi-supervised greedy algorithm , 2013, Neural Networks.

[23]  Cheng Wu,et al.  Semi-Supervised and Unsupervised Extreme Learning Machines , 2014, IEEE Transactions on Cybernetics.

[24]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[25]  Yong Shi,et al.  Laplacian twin support vector machine for semi-supervised classification , 2012, Neural Networks.

[26]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[27]  Giorgio Valentini,et al.  A neural network algorithm for semi-supervised node label learning from unbalanced data , 2013, Neural Networks.

[28]  Estevam R. Hruschka,et al.  Using unsupervised information to improve semi-supervised tweet sentiment classification , 2016, Inf. Sci..

[29]  Yan Yang,et al.  Driver Distraction Detection Using Semi-Supervised Machine Learning , 2016, IEEE Transactions on Intelligent Transportation Systems.

[30]  Qingming Huang,et al.  Online web video topic detection and tracking with semi-supervised learning , 2014, Multimedia Systems.

[31]  Alejandro Figueroa,et al.  Leveraging linguistic traits and semi-supervised learning to single out informational content across how-to community question-answering archives , 2017, Inf. Sci..

[32]  Mikhail Belkin,et al.  Semi-Supervised Learning on Riemannian Manifolds , 2004, Machine Learning.

[33]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[34]  Wei Liu,et al.  Multi-Modal Curriculum Learning for Semi-Supervised Image Classification , 2016, IEEE Transactions on Image Processing.

[35]  Thorsten Joachims,et al.  Transductive Support Vector Machines , 2006, Semi-Supervised Learning.

[36]  Simone Scardapane,et al.  Fully Decentralized Semi-supervised Learning via Privacy-preserving Matrix Completion , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Weisheng Chen,et al.  Distributed cooperative learning algorithms using wavelet neural network , 2017, Neural Computing and Applications.