Sparse Multi-Prototype Classification

We present a new class of sparse multi-prototype classifiers, designed to combine the computational advantages of sparse predictors with the non-linear power of prototype-based classification techniques. This combination makes sparse multi-prototype models especially well-suited for resource constrained computational platforms, such as those found in IoT devices. We cast our supervised learning problem as a convex-concave saddle point problem and design a provably-fast algorithm to solve it. We complement our theoretical analysis with an empirical study that demonstrates the power of our methodology.

[1]  Shuicheng Yan,et al.  Auto-Grouped Sparse Representation for Visual Analysis , 2012, ECCV.

[2]  Prateek Jain,et al.  ProtoNN: Compressed and Accurate kNN for Resource-scarce Devices , 2017, ICML.

[3]  Yoram Singer,et al.  The Forgetron: A Kernel-Based Perceptron on a Budget , 2008, SIAM J. Comput..

[4]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[5]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[6]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[7]  Yoram Singer,et al.  Support Vector Machines on a Budget , 2006, NIPS.

[8]  Yoram Singer,et al.  Efficient projections onto the {\it l}$_{\mbox{1}}$-ball for learning in high dimensions , 2008, ICML 2008.

[9]  Sanjiv Kumar,et al.  Fast Classification with Binary Prototypes , 2017, AISTATS.

[10]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[11]  Bingsheng He,et al.  An Algorithmic Framework of Generalized Primal–Dual Hybrid Gradient Methods for Saddle Point Problems , 2017, Journal of Mathematical Imaging and Vision.

[12]  Stephen Tyree,et al.  Stochastic Neighbor Compression , 2014, ICML.

[13]  Panos M. Pardalos,et al.  An algorithm for a singly constrained class of quadratic programs subject to upper and lower bounds , 1990, Math. Program..

[14]  Rohit Prabhavalkar,et al.  Compressing deep neural networks using a rank-constrained topology , 2015, INTERSPEECH.

[15]  Venkatesh Saligrama,et al.  Pruning Random Forests for Prediction on a Budget , 2016, NIPS.

[16]  Koby Crammer,et al.  On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines , 2002, J. Mach. Learn. Res..

[17]  Ebru Arisoy,et al.  Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[18]  Jason Weston,et al.  Support vector machines for multi-class pattern recognition , 1999, ESANN.

[19]  Jianxin Wu,et al.  ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[20]  P. Brucker.  Review of recent development: An O( n) algorithm for quadratic knapsack problems , 1984 .

[21]  Yuchen Zhang,et al.  Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimization , 2014, ICML.

[22]  Saurabh Goyal,et al.  Resource-efficient Machine Learning in 2 KB RAM for the Internet of Things , 2017, ICML.

[23]  Francis R. Bach,et al.  Clusterpath: an Algorithm for Clustering using Convex Fusion Penalties , 2011, ICML.

[24]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[25]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[26]  Alessandro Sperduti,et al.  Multiclass Classification with Multi-Prototype Support Vector Machines , 2005, J. Mach. Learn. Res..

[27]  Sinno Jialin Pan,et al.  Communication-Efficient Distributed Primal-Dual Algorithm for Saddle Point Problem , 2017, UAI.

[28]  Yixin Chen,et al.  Compressing Neural Networks with the Hashing Trick , 2015, ICML.

[29]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.