Stochastic Fixed Point Optimization Algorithm for Classifier Ensemble

This paper considers a classifier ensemble problem with sparsity and diversity learning, which arises in the field of machine learning, and shows that the classifier ensemble problem can be formulated as a convex stochastic optimization problem over the fixed point set of a quasi-nonexpansive mapping. Specifically, for such a problem, this paper proposes an algorithm referred to as the stochastic fixed point optimization algorithm and performs a convergence analysis for three types of step size: 1) constant step size; 2) decreasing step size; and 3) a step size computed by line searches. In the case of a constant step size, the results indicate that a sufficiently small constant step size allows a solution to the problem to be approximated. In the case of a decreasing step size, conditions are shown under which the algorithm converges in probability to a solution. For the third case, a variation of the basic proposed algorithm also achieves convergence in probability to a solution. The high classification accuracies of the proposed algorithms are demonstrated through numerical comparisons with the conventional algorithm.

[1]  Hideaki Iiduka,et al.  Convergence analysis of iterative methods for nonsmooth convex optimization over fixed point sets of quasi-nonexpansive mappings , 2015, Mathematical Programming.

[2]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[3]  Hideaki Iiduka,et al.  Fixed Point Optimization Algorithms for Distributed Optimization in Networked Systems , 2013, SIAM J. Optim..

[4]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[5]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[6]  Adrian S. Lewis,et al.  Nonsmooth optimization via quasi-Newton methods , 2012, Mathematical Programming.

[7]  J. Borwein,et al.  Convex Analysis And Nonlinear Optimization , 2000 .

[8]  V. V. Vasin,et al.  Ill-posed problems with a priori information , 1995 .

[9]  Zili Zhang,et al.  Sample Subset Optimization Techniques for Imbalanced and Ensemble Learning Problems in Bioinformatics Applications , 2014, IEEE Transactions on Cybernetics.

[10]  Angelia Nedic,et al.  Incremental Stochastic Subgradient Algorithms for Convex Optimization , 2008, SIAM J. Optim..

[11]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[12]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[13]  Patrick L. Combettes,et al.  A block-iterative surrogate constraint splitting method for quadratic signal recovery , 2003, IEEE Trans. Signal Process..

[14]  Saeed Ghadimi,et al.  Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization, II: Shrinking Procedures and Optimal Algorithms , 2013, SIAM J. Optim..

[15]  Hideaki Iiduka,et al.  Proximal point algorithms for nonsmooth convex optimization with fixed point constraints , 2016, Eur. J. Oper. Res..

[16]  Kaizhu Huang,et al.  A novel classifier ensemble method with sparsity and diversity , 2014, Neurocomputing.

[17]  Li Zhang,et al.  Sparse ensembles using weighted combination methods based on linear programming , 2011, Pattern Recognit..

[18]  I. Yamada.  The Hybrid Steepest Descent Method for the Variational Inequality Problem over the Intersection of Fixed Point Sets of Nonexpansive Mappings , 2001 .

[19]  Yoichi Hayashi,et al.  Optimality and convergence for convex ensemble learning with sparsity and diversity based on fixed point optimization , 2018, Neurocomputing.

[20]  Saeed Ghadimi,et al.  Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework , 2012, SIAM J. Optim..

[21]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[22]  Kaizhu Huang,et al.  Convex ensemble learning with sparsity and diversity , 2014, Inf. Fusion.

[23]  Heinz H. Bauschke,et al.  A Weak-to-Strong Convergence Principle for Fejé-Monotone Methods in Hilbert Spaces , 2001, Math. Oper. Res..

[24]  Minjie Zhang,et al.  Collective Learning for the Emergence of Social Norms in Networked Multiagent Systems , 2014, IEEE Transactions on Cybernetics.

[25]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[26]  Heinz H. Bauschke,et al.  A projection method for approximating fixed points of quasi nonexpansive mappings without the usual demiclosedness condition , 2012, 1211.1639.

[27]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[28]  Paul-Emile Maingé,et al.  The viscosity approximation process for quasi-nonexpansive mappings in Hilbert spaces , 2010, Comput. Math. Appl..

[29]  Pedro Antonio Gutiérrez,et al.  Projection-Based Ensemble Learning for Ordinal Regression , 2014, IEEE Transactions on Cybernetics.

[30]  Alexander Shapiro,et al.  Lectures on Stochastic Programming: Modeling and Theory , 2009 .

[31]  Terry Windeatt,et al.  Pruning of Error Correcting Output Codes by optimization of accuracy–diversity trade off , 2014, Machine Learning.

[32]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[33]  V. Borkar.  Stochastic Approximation: A Dynamical Systems Viewpoint , 2008 .

[34]  W. Marsden.  I and J , 2012 .

[35]  William Nick Street,et al.  Ensemble Pruning Via Semi-definite Programming , 2006, J. Mach. Learn. Res..