Hybrid Algorithms with Instance-Based Classification

In this paper we aim to show that instance-based classification can replace the classifier component of a rule learner and of maximum-entropy modeling, thereby improving the generalization accuracy of both algorithms. We describe hybrid algorithms that combine rule learning models and maximum-entropy modeling with instance-based classification. Experimental results show that both hybrids are able to outperform the parent algorithm. We analyze and compare the overlap in errors and the statistical bias and variance of the hybrids, their parent algorithms, and a plain instance-based learner. We observe that the successful hybrid algorithms have a lower statistical bias component in the error than their parent algorithms; the fewer errors they make are also less systematic.

[1]  Stefan Wess,et al.  Topics in Case-Based Reasoning , 1994 .

[2]  Walter Daelemans,et al.  TiMBL: Tilburg Memory-Based Learner , 2007 .

[3]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[4]  S. Salzberg,et al.  A weighted nearest neighbor algorithm for learning with symbolic features , 2004, Machine Learning.

[5]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[6]  Michèle Sebag,et al.  A Rule-Based Similarity Measure , 1993, EWCBR.

[7]  Eric Brill,et al.  Classifier Combination for Improved Lexical Disambiguation , 1998, ACL.

[8]  Iris Hendrickx,et al.  Maximum-entropy parameter estimation for the k-NN modified value-difference kernel , 2005 .

[9]  Silviu Guiasu,et al.  The principle of maximum entropy , 1985 .

[10]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[11]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[12]  .. M. Ting,et al.  EXPLORING A FRAMEWORK FOR INSTANCE BASEDLEARNING AND NAIVE BAYESIAN CLASSIFIERSK , 1994 .

[13]  Antal van den Bosch,et al.  Feature transformation through rule induction : A case study with the k-NN classifier , 2004 .

[14]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[15]  Usama M. Fayyad,et al.  Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning , 1993, IJCAI.

[16]  David W. Aha,et al.  A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms , 1997, Artificial Intelligence Review.

[17]  Walter Daelemans,et al.  TiMBL: Tilburg Memory-Based Learner, version 2.0, Reference guide , 1998 .

[18]  Pedro M. Domingos.  Unifying Instance-Based and Rule-Based Induction , 1996, Machine Learning.

[19]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[20]  Antal van den Bosch.  Wrapped progressive sampling search for optimizing learning algorithm parameters , 2005 .

[21]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .