Improving Tree augmented Naive Bayes for class probability estimation

Numerous algorithms have been proposed to improve Naive Bayes (NB) by weakening its conditional attribute independence assumption, among which Tree Augmented Naive Bayes (TAN) has demonstrated remarkable classification performance in terms of classification accuracy or error rate, while maintaining efficiency and simplicity. In many real-world applications, however, classification accuracy or error rate is not enough. For example, in direct marketing, we often need to deploy different promotion strategies to customers with different likelihood (class probability) of buying some products. Thus, accurate class probability estimation is often required to make optimal decisions. In this paper, we investigate the class probability estimation performance of TAN in terms of conditional log likelihood (CLL) and present a new algorithm to improve its class probability estimation performance by the spanning TAN classifiers. We call our improved algorithm Averaged Tree Augmented Naive Bayes (ATAN). The experimental results on a large number of UCI datasets published on the main web site of Weka platform show that ATAN significantly outperforms TAN and all the other algorithms used to compare in terms of CLL.

[1]  Ian Witten,et al.  Data Mining , 2000 .

[2]  Doug Fisher,et al.  Learning from Data: Artificial Intelligence and Statistics V , 1996 .

[3]  S. Appavu alias Balamurugan,et al.  NB+: An improved Naïve Bayesian algorithm , 2011, Knowl. Based Syst..

[4]  David Maxwell Chickering,et al.  Learning Bayesian Networks is , 1994 .

[5]  Geoffrey I. Webb,et al.  Not So Naive Bayes: Aggregating One-Dependence Estimators , 2005, Machine Learning.

[6]  Liangxiao Jiang,et al.  A Novel Bayes Model: Hidden Naive Bayes , 2009, IEEE Transactions on Knowledge and Data Engineering.

[7]  Ning Zhong,et al.  Developing Mining-Grid Centric E-Finance Portals for Risk Management and Decision Making , 2007, Int. J. Pattern Recognit. Artif. Intell..

[8]  Liangxiao Jiang,et al.  Learning Naive Bayes for Probability Estimation by Feature Selection , 2006, Canadian Conference on AI.

[9]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[10]  Liangxiao Jiang,et al.  Weightily averaged one-dependence estimators , 2006 .

[11]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[12]  Foster J. Provost,et al.  Active Sampling for Class Probability Estimation and Ranking , 2004, Machine Learning.

[13]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[14]  Pedro M. Domingos,et al.  Learning Bayesian network classifiers by maximizing conditional likelihood , 2004, ICML.

[15]  Eamonn J. Keogh,et al.  Learning augmented Bayesian classifiers: A comparison of distribution-based and classification-based approaches , 1999, AISTATS.

[16]  Stan Matwin,et al.  Discriminative parameter learning for Bayesian networks , 2008, ICML '08.

[17]  Liangxiao Jiang,et al.  Decision Tree with Better Class Probability Estimation , 2009, Int. J. Pattern Recognit. Artif. Intell..

[18]  Russell Greiner,et al.  Discriminative Model Selection for Belief Net Structures , 2005, AAAI.

[19]  Pedro M. Domingos,et al.  Tree Induction for Probability-Based Ranking , 2003, Machine Learning.

[20]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[21]  Michael G. Madden,et al.  On the classification performance of TAN and general Bayesian networks , 2008, Knowl. Based Syst..

[22]  Charles X. Ling,et al.  Toward Bayesian Classifiers with Accurate Probabilities , 2002, PAKDD.

[23]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[24]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[25]  Liangxiao Jiang,et al.  Learning decision tree for ranking , 2009, Knowledge and Information Systems.

[26]  Charles X. Ling,et al.  An Improved Learning Algorithm for Augmented Naive Bayes , 2001, PAKDD.

[27]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[28]  Xiaoyi Jiang,et al.  Structure identification of Bayesian classifiers based on GMDH , 2009, Knowl. Based Syst..