Matrixized learning machine with modified pairwise constraints

Matrix-pattern-oriented Classifier Design (MatCD) has been demonstrated to be effective in terms of the classification performance since it utilizes two-sided weight vectors to constrain the matrix-based patterns. However, the existing MatCD might not be able to acquire the prior distribution knowledge, such as the relationship between two patterns. Inspired by the Pairwise Constraints (PC) method, i.e., must-links and cannot-links between the patterns, this paper introduces a new regularization term named Rp with a modified PC method into MatCD. The new classifier design strategy is expected to not only learn the structural information of each pattern itself, but also acquire the prior distribution knowledge about each constrained pair with both the discrimination metric from the traditional PC and the spatial distance measure from the heat kernel method. In practice, this paper selects one typical matrixized classifier named MatMHKS as the basic building block and introduces the term Rp into it. The newly-proposed classifier is named MLMMPC and the subsequent experiments validate the effectiveness of it. Two major contributions of this paper can be concluded as (1) improving the existing matrix-pattern-oriented classifier design techniques and (2) modifying the traditional PC method by combining the discrimination metric and the distance measure together. HighlightsA novel matrix-oriented classification algorithm named MLMMPC is proposed.To improve the original matrix learning framework by a new regularization term R p .To combine pairwise constraints and spatial measure together in R p .Implementability is demonstrated on both image- and vector-based datasets.

[1]  Pascal Denis,et al.  Fast Gaussian Pairwise Constrained Spectral Clustering , 2014, ECML/PKDD.

[2]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[3]  Daoqiang Zhang,et al.  (2D)2PCA: Two-directional two-dimensional PCA for efficient face representation and recognition , 2005, Neurocomputing.

[4]  Xinbo Gao,et al.  Semi-supervised Gaussian process latent variable model with pairwise constraints , 2010, Neurocomputing.

[5]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[6]  Zhe Wang,et al.  IMAT: matrix learning machine with interpolation mapping , 2014 .

[7]  Rich Caruana,et al.  Improving Classification with Pairwise Constraints: A Margin-Based Approach , 2008, ECML/PKDD.

[8]  Feiping Nie,et al.  Multiple rank multi-linear SVM for matrix data classification , 2014, Pattern Recognit..

[9]  Daoqiang Zhang,et al.  Pattern Representation in Feature Extraction and Classifier Design: Matrix Versus Vector , 2008, IEEE Transactions on Neural Networks.

[10]  Ja-Chen Lin,et al.  A new LDA-based face recognition system which can solve the small sample size problem , 1998, Pattern Recognit..

[11]  Dana Ron,et al.  An experimental and theoretical comparison of model selection methods , 1995, COLT '95.

[12]  John Shawe-Taylor,et al.  Two view learning: SVM-2K, Theory and Practice , 2005, NIPS.

[13]  Vladimir Koltchinskii,et al.  Rademacher penalties and structural risk minimization , 2001, IEEE Trans. Inf. Theory.

[14]  Peter L. Bartlett,et al.  Model Selection and Error Estimation , 2000, Machine Learning.

[15]  Alejandro F. Frangi,et al.  Two-dimensional PCA: a new approach to appearance-based face representation and recognition , 2004 .

[16]  Luiz Eduardo Soares de Oliveira,et al.  Pairwise fusion matrix for combining classifiers , 2007, Pattern Recognit..

[17]  Daoqiang Zhang,et al.  Bagging Constraint Score for feature selection with pairwise constraints , 2010, Pattern Recognit..

[18]  Tianrui Li,et al.  Constraint projections for semi-supervised affinity propagation , 2012, Knowl. Based Syst..

[19]  YangJian,et al.  Two-Dimensional PCA , 2004 .

[20]  Jacek M. Łȩski,et al.  Ho--Kashyap classifier with generalization control , 2003 .

[21]  Yi-Zeng Liang,et al.  Monte Carlo cross validation , 2001 .

[22]  Ming Yang,et al.  A novel hypothesis-margin based approach for feature selection with side pairwise constraints , 2010, Neurocomputing.

[23]  Xuelong Li,et al.  Supervised Tensor Learning , 2005, ICDM.

[24]  Xiao-Jun Wu,et al.  A new semi-supervised clustering algorithm with pairwise constraints by competitive agglomeration , 2011, Appl. Soft Comput..

[25]  Palaiahnakote Shivakumara,et al.  (2D)2LDA: An efficient approach for face recognition , 2006, Pattern Recognit..

[26]  Peter L. Bartlett,et al.  Rademacher and Gaussian Complexities: Risk Bounds and Structural Results , 2003, J. Mach. Learn. Res..

[27]  Xindong Wu,et al.  Extracting elite pairwise constraints for clustering , 2013, Neurocomputing.

[28]  Marco Maggini,et al.  Learning from pairwise constraints by Similarity Neural Networks , 2012, Neural Networks.

[29]  Stephen J. Wright,et al.  Dissimilarity in Graph-Based Semi-Supervised Classification , 2007, AISTATS.

[30]  Xiangyang Xue,et al.  An efficient Kernel-based matrixized least squares support vector machine , 2011, Neural Computing and Applications.

[31]  Qiang Qian,et al.  Multi-view classification with cross-view must-link and cannot-link side information , 2013, Knowl. Based Syst..

[32]  John M. Lee.  Introduction to Smooth Manifolds , 2002 .

[33]  Aiguo Song,et al.  Improving clustering with pairwise constraints: a discriminative approach , 2012, Knowledge and Information Systems.

[34]  Songcan Chen,et al.  New Least Squares Support Vector Machines Based on Matrix Patterns , 2007, Neural Processing Letters.

[35]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[36]  Ming Li,et al.  2D-LDA: A statistical linear discriminant analysis for image matrix , 2005, Pattern Recognit. Lett..

[37]  Qi Huang,et al.  Semi-supervised fuzzy clustering with metric learning and entropy regularization , 2012, Knowl. Based Syst..

[38]  Yuxiao Hu,et al.  Learning a Spatially Smooth Subspace for Face Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[40]  Tomaso Poggio,et al.  Image Representations for Visual Learning , 1996, Science.

[41]  Nenghai Yu,et al.  Maximum Margin Clustering with Pairwise Constraints , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[42]  Martial Hebert,et al.  A spectral technique for correspondence problems using pairwise constraints , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[43]  Xiaojun Wu,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Joachim M. Buhmann,et al.  Learning with constrained and unlabelled data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[45]  Hui Xue,et al.  Glocalization pursuit support vector machine , 2011, Neural Computing and Applications.

[46]  Charu C. Aggarwal,et al.  Towards graphical models for text processing , 2012, Knowledge and Information Systems.

[47]  Wenwen Liu,et al.  Multi-view learning with Universum , 2014, Knowl. Based Syst..

[48]  Jim Jing-Yan Wang,et al.  Multiple graph regularized nonnegative matrix factorization , 2013, Pattern Recognit..

[49]  Miguel Á. Carreira-Perpiñán,et al.  Constrained spectral clustering through affinity propagation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Michael I. Jordan,et al.  Distance Metric Learning with Application to Clustering with Side-Information , 2002, NIPS.

[51]  Daoqiang Zhang,et al.  Constraint Score: A new filter method for feature selection with pairwise constraints , 2008, Pattern Recognit..

[52]  Victor L. Brailovsky,et al.  On global, local, mixed and neighborhood kernels for support vector machines , 1999, Pattern Recognit. Lett..

[53]  Songcan Chen,et al.  Matrix-pattern-oriented Ho-Kashyap classifier with regularization learning , 2007, Pattern Recognit..

[54]  Jun Yu,et al.  Pairwise constraints based multiview features fusion for scene classification , 2013, Pattern Recognit..

[55]  R. L. Kashyap,et al.  An Algorithm for Linear Inequalities and its Applications , 1965, IEEE Trans. Electron. Comput..

[56]  Min Wu,et al.  Multi-label ensemble based on variable pairwise constraint projection , 2013, Inf. Sci..

[57]  Ivor W. Tsang,et al.  Efficient kernel feature extraction for massive data sets , 2006, KDD '06.

[58]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[59]  Changshui Zhang,et al.  Boosting with pairwise constraints , 2010, Neurocomputing.

[60]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[61]  Christian Bauckhage.  Robust Tensor Classifiers for Color Object Recognition , 2007, ICIAR.