LoGID: An adaptive framework combining local and global incremental learning for dynamic selection of ensembles of HMMs

In this work, we propose the LoGID (Local and Global Incremental Learning for Dynamic Selection) framework, the main goal of which is to adapt hidden Markov model-based pattern recognition systems during both the generalization and learning phases. Given that the baseline system is composed of a pool of base classifiers, adaptation during generalization is performed through the dynamic selection of the members of this pool that best recognize each test sample. This is achieved by the proposed K-nearest output profiles algorithm, while adaptation during learning consists of gradually updating the knowledge embedded in the base classifiers, by processing previously unobserved data. This phase employs two types of incremental learning: local and global. Local incremental learning involves updating the pool of base classifiers by adding new members to this set. The new members are created with the Learn++ algorithm. Global incremental learning, in contrast, consists of updating the set of output profiles used during generalization. The proposed framework has been evaluated on a diversified set of databases. The results indicate that LoGID is promising. For most databases, the recognition rates achieved by the proposed method are higher than those achieved by other state-of-the-art approaches, such as batch learning. Furthermore, the simulated incremental learning setting demonstrates that LoGID can effectively improve the performance of systems created with small training sets as more data are observed over time.

[1]  Maneesha Singh,et al.  A dynamic classifier selection and combination approach to image region labelling , 2005, Signal Process. Image Commun..

[2]  Eiji Takimoto,et al.  On-Line Estimation of Hidden Markov Model Parameters , 2000, Discovery Science.

[3]  Xue Wang DURATIONALLY CONSTRAINED TRAINING OF HMM WITHOUT EXPLICIT STATE DURATIONAL PDF , 1994 .

[4]  Luca Maria Gambardella,et al.  Convolutional Neural Network Committees for Handwritten Character Classification , 2011, 2011 International Conference on Document Analysis and Recognition.

[5]  Susan M. Bridges,et al.  Incremental Estimation of Discrete Hidden Markov Models Based on a New Backward Procedure , 2005, AAAI.

[6]  Robert Sabourin,et al.  Dynamic selection approaches for multiple classifier systems , 2011, Neural Computing and Applications.

[7]  Heng Tao Shen,et al.  Exploring Bit-Difference for Approximate KNN Search in High-dimensional Databases , 2005, ADC.

[8]  Flávio Bortolozzi,et al.  The recognition of handwritten numeral strings using a two-stage HMM-based method , 2003, International Journal on Document Analysis and Recognition.

[9]  Anne M. P. Canuto,et al.  Using Accuracy and Diversity to Select Classifiers to Build Ensembles , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[10]  Robert Sabourin,et al.  Ensemble of HMM classifiers based on the clustering validity index for a handwritten numeral recognizer , 2009, Pattern Analysis and Applications.

[11]  Ludmila I. Kuncheva,et al.  Clustering-and-selection model for classifier combination , 2000, KES'2000. Fourth International Conference on Knowledge-Based Intelligent Engineering Systems and Allied Technologies. Proceedings (Cat. No.00TH8516).

[12]  L. K. Hansen,et al.  The Error-Reject Tradeoff , 1997 .

[13]  Gianluigi Mongillo,et al.  Online Learning with Hidden Markov Models , 2008, Neural Computation.

[14]  Fabio Roli,et al.  Dynamic classifier selection based on multiple classifier behaviour , 2001, Pattern Recognit..

[15]  Robert Sabourin,et al.  “One Against One” or “One Against All”: Which One is Better for Handwriting Recognition with SVMs? , 2006 .

[16]  Robert Sabourin,et al.  Leave-One-Out-Training and Leave-One-Out-Testing Hidden Markov Models for a Handwritten Numeral Recognizer: The Implications of a Single Classifier and Multiple Classifications , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  James C. Bezdek,et al.  Decision templates for multiple classifier fusion: an experimental comparison , 2001, Pattern Recognit..

[18]  Robert Sabourin,et al.  A dynamic overproduce-and-choose strategy for the selection of classifier ensembles , 2008, Pattern Recognit..

[19]  Vasant Honavar,et al.  Learn++: an incremental learning algorithm for supervised neural networks , 2001, IEEE Trans. Syst. Man Cybern. Part C.

[20]  Nojun Kwak,et al.  Feature extraction for classification problems and its application to face recognition , 2008, Pattern Recognit..

[21]  Yi-Ming Chen,et al.  Combining incremental Hidden Markov Model and Adaboost algorithm for anomaly intrusion detection , 2009, CSI-KDD '09.

[22]  Nacereddine Hammami,et al.  Improved tree model for arabic speech recognition , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[23]  Robi Polikar,et al.  Learn$^{++}$ .NC: Combining Ensemble of Classifiers With Dynamically Weighted Consult-and-Vote for Efficient Incremental Learning of New Classes , 2009, IEEE Transactions on Neural Networks.

[24]  Xindong Wu,et al.  Dynamic classifier selection for effective mining from noisy data streams , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[25]  Robert Sabourin,et al.  Dynamic Ensemble Selection for Off-Line Signature Verification , 2011, MCS.

[26]  Robert Sabourin,et al.  From dynamic classifier selection to dynamic ensemble selection , 2008, Pattern Recognit..

[27]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[28]  Robert Sabourin,et al.  Evaluation of incremental learning algorithms for HMM in the recognition of alphanumeric characters , 2009, Pattern Recognit..

[29]  R. Polikar,et al.  Dynamically weighted majority voting for incremental learning and comparison of three boosting based approaches , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[30]  Lior Rokach,et al.  Ensemble-based classifiers , 2010, Artificial Intelligence Review.

[31]  Kevin W. Bowyer,et al.  Combination of Multiple Classifiers Using Local Accuracy Estimates , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[33]  Ethem Alpaydin,et al.  Incremental construction of classifier and discriminant ensembles , 2009, Inf. Sci..

[34]  Abdel Ghaffmr Mokamed Ahmed,et al.  One Against All , 2009 .

[35]  Robert Sabourin,et al.  An Evaluation of Over-Fit Control Strategies for Multi-Objective Evolutionary Optimization , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[36]  Robert Sabourin,et al.  Dynamic Selection of Ensembles of Classifiers Using Contextual Information , 2010, MCS.

[37]  Robert Sabourin,et al.  Adaptive Incremental Learning with an Ensemble of Support Vector Machines , 2010, 2010 20th International Conference on Pattern Recognition.