Incremental Import Vector Machines for Classifying Hyperspectral Data

In this paper, we propose an incremental learning strategy for import vector machines (IVM), which is a sparse kernel logistic regression approach. We use the procedure for the concept of self-training for sequential classification of hyperspectral data. The strategy comprises the inclusion of new training samples to increase the classification accuracy and the deletion of noninformative samples to be memory and runtime efficient. Moreover, we update the parameters in the incremental IVM model without retraining from scratch. Therefore, the incremental classifier is able to deal with large data sets. The performance of the IVM in comparison to support vector machines (SVM) is evaluated in terms of accuracy, and experiments are conducted to assess the potential of the probabilistic outputs of the IVM. Experimental results demonstrate that the IVM and SVM perform similar in terms of classification accuracy. However, the number of import vectors is significantly lower when compared to the number of support vectors, and thus, the computation time during classification can be decreased. Moreover, the probabilities provided by IVM are more reliable, when compared to the probabilistic information, derived from an SVM's output. In addition, the proposed self-training strategy can increase the classification accuracy. Overall, the IVM and its incremental version is worthwhile for the classification of hyperspectral data.

[1]  Shaoning Pang,et al.  Incremental linear discriminant analysis for classification of data streams , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  S. Sathiya Keerthi,et al.  A Fast Dual Algorithm for Kernel Logistic Regression , 2002, 2007 International Joint Conference on Neural Networks.

[3]  Gavin C. Cawley,et al.  Efficient model selection for kernel logistic regression , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[4]  Paul M. Mather,et al.  Some issues in the classification of DAIS hyperspectral data , 2006 .

[5]  Francesca Bovolo,et al.  Semisupervised One-Class Support Vector Machines for Classification of Remote Sensing Data , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Claire Cardie,et al.  Weakly Supervised Natural Language Learning Without Redundant Views , 2003, NAACL.

[7]  Patrick Hostert,et al.  Classifying segmented hyperspectral data from a heterogeneous urban environment using support vector machines , 2007 .

[8]  Ji Zhu,et al.  Kernel Logistic Regression and the Import Vector Machine , 2001, NIPS.

[9]  Antonio J. Plaza,et al.  Hyperspectral Image Segmentation Using a New Bayesian Approach With Active Learning , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Neil D. Lawrence,et al.  Fast Sparse Gaussian Process Methods: The Informative Vector Machine , 2002, NIPS.

[11]  Mário A. T. Figueiredo.  Adaptive Sparseness for Supervised Learning , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Jon Atli Benediktsson,et al.  SVM- and MRF-Based Method for Accurate Classification of Hyperspectral Images , 2010, IEEE Geoscience and Remote Sensing Letters.

[13]  Lawrence Carin,et al.  Sparse multinomial logistic regression: fast algorithms and generalization bounds , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Pramod K. Varshney,et al.  Logistic Regression for Feature Selection and Soft Classification of Remote Sensing Data , 2006, IEEE Geoscience and Remote Sensing Letters.

[15]  Peter J. Rousseeuw,et al.  Robust Regression and Outlier Detection , 2005, Wiley Series in Probability and Statistics.

[16]  Johannes R. Sveinsson,et al.  Classification of hyperspectral data from urban areas based on extended morphological profiles , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[17]  Pao-Ta Yu,et al.  A Dynamic Subspace Method for Hyperspectral Image Classification , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[19]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[20]  Giles M. Foody,et al.  RVM‐based multi‐class classification of remotely sensed data , 2008 .

[21]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[22]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[23]  Giles M. Foody,et al.  Multiclass and Binary SVM Classification: Implications for Training and Classification Users , 2008, IEEE Geoscience and Remote Sensing Letters.

[24]  Ichiro Takeuchi,et al.  Multiple Incremental Decremental Learning of Support Vector Machines , 2009, IEEE Transactions on Neural Networks.

[25]  Silvia Scarpetta,et al.  Uncertainty Analysis for the Classification of Multispectral Satellite Images Using SVMs and SOMs , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Mikhail F. Kanevski,et al.  A Survey of Active Learning Algorithms for Supervised Remote Sensing Image Classification , 2011, IEEE Journal of Selected Topics in Signal Processing.

[27]  T. Warner,et al.  Integrating visible, near-infrared and short-wave infrared hyperspectral and multispectral thermal imagery for geological mapping at Cuprite, Nevada , 2007 .

[28]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[29]  Ping Zhong,et al.  Learning Conditional Random Fields for Classification of Hyperspectral Images , 2010, IEEE Transactions on Image Processing.

[30]  Gustavo Camps-Valls,et al.  Spatio-Spectral Remote Sensing Image Classification With Graph Kernels , 2010, IEEE Geoscience and Remote Sensing Letters.

[31]  Peng Zhang,et al.  Dynamic Learning of SMLR for Feature Selection and Classification of Hyperspectral Data , 2008, IEEE Geoscience and Remote Sensing Letters.

[32]  Jon Atli Benediktsson,et al.  Sensitivity of Support Vector Machines to Random Feature Selection in Classification of Hyperspectral Data , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[33]  Nathalie Japkowicz,et al.  The Class Imbalance Problem: Significance and Strategies , 2000 .

[34]  Glenn Fung,et al.  Incremental Support Vector Machine Classification , 2002, SDM.

[35]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote sensing images with support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Begüm Demir,et al.  Hyperspectral Image Classification Using Relevance Vector Machines , 2007, IEEE Geoscience and Remote Sensing Letters.

[37]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[38]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[40]  José M. Bioucas-Dias,et al.  Bayesian Hyperspectral Image Segmentation With Discriminative Class Learning , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[42]  Gavin C. Cawley,et al.  Sparse Multinomial Logistic Regression via Bayesian L1 Regularisation , 2006, NIPS.

[43]  Alexander F. H. Goetz,et al.  Three decades of hyperspectral remote sensing of the Earth: a personal view. , 2009 .

[44]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[45]  Luis Guanter,et al.  Simulation of Optical Remote-Sensing Scenes With Application to the EnMAP Hyperspectral Mission , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Sanjiv Kumar,et al.  Discriminative Random Fields , 2006, International Journal of Computer Vision.

[47]  Lehel Csató,et al.  Sparse On-Line Gaussian Processes , 2002, Neural Computation.

[48]  John A. Richards,et al.  Analysis of remotely sensed data: the formative decades and the future , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[49]  Giles M. Foody,et al.  Feature Selection for Classification of Hyperspectral Data by SVM , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[50]  S. Weisberg,et al.  Residuals and Influence in Regression , 1982 .

[51]  J. R. Sveinsson,et al.  Mapping of hyperspectral AVIRIS data using machine-learning algorithms , 2009 .

[53]  Jon Atli Benediktsson,et al.  Recent Advances in Techniques for Hyperspectral Image Processing , 2009 .

[54]  Anil K. Jain,et al.  Bayesian learning of sparse classifiers , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[55]  Pietro Perona,et al.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[56]  Nello Cristianini,et al.  Large Margin DAGs for Multiclass Classification , 1999, NIPS.

[57]  Ioannis Z. Gitas,et al.  Mapping Postfire Vegetation Recovery Using EO-1 Hyperion Imagery , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[58]  Ping Zhong,et al.  Learning Sparse CRFs for Feature Selection and Classification of Hyperspectral Imagery , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[59]  Joydeep Ghosh,et al.  An Active Learning Approach to Hyperspectral Data Classification , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[60]  Antonio J. Plaza,et al.  Semisupervised Hyperspectral Image Segmentation Using Multinomial Logistic Regression With Active Learning , 2010, IEEE Transactions on Geoscience and Remote Sensing.