Efficient Online Classification Using an Ensemble of Bayesian Linear Logistic Regressors

We present a novel ensemble of logistic linear regressors that combines the robustness of online Bayesian learning with the flexibility of ensembles. The ensemble of classifiers are built on top of a Randomly Varying Coefficient model designed for online regression with the fusion of classifiers done at the level of regression before converting it into a class label using a logistic link function. The locally weighted logistic regressor is compared against the state-of-the-art methods to reveal its excellent generalization performance with low time and space complexities.

[1]  A. Wayne Whitney,et al.  A Direct Method of Nonparametric Measurement Selection , 1971, IEEE Transactions on Computers.

[2]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[3]  Josef Kittler,et al.  Floating search methods in feature selection , 1994, Pattern Recognit. Lett..

[4]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[5]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[6]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[7]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Chris Cox,et al.  Multiple-criteria genetic algorithms for feature selection in neuro-fuzzy modeling , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[9]  Pavel Pudil,et al.  Oscillating search algorithms for feature selection , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[10]  Volker Tresp,et al.  A Bayesian Committee Machine , 2000, Neural Computation.

[11]  Horst Bunke,et al.  Feature selection algorithms for the generation of multiple classifier systems and their application to handwritten word recognition , 2004 .

[12]  Stuart J. Russell,et al.  Online bagging and boosting , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[13]  Sarunas Raudys.  Feature Over-Selection , 2006, SSPR/SPR.

[14]  Stefan Schaal,et al.  Kernel Carpentry for Online Regression Using Randomly Varying Coefficient Model , 2007, IJCAI.

[15]  Ludmila I. Kuncheva,et al.  A stability index for feature selection , 2007, Artificial Intelligence and Applications.

[16]  Ting Chen,et al.  Ensemble Feature Selection: Consistent Descriptor Subsets for Multiple QSAR Models , 2007, J. Chem. Inf. Model..

[17]  Murat Dundar,et al.  Bayesian multiple instance learning: automatic feature selection and inductive transfer , 2008, ICML '08.

[18]  Pavel Pudil,et al.  Dynamic Oscillating Search algorithm for feature selection , 2008, 2008 19th International Conference on Pattern Recognition.

[19]  Jana Novovicová,et al.  Evaluating the Stability of Feature Selectors That Optimize Feature Subset Cardinality , 2008, SSPR/SPR.

[20]  Y. Saeys,et al.  Towards robust feature selection techniques , 2008 .

[21]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[22]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.