Scalable Structure Learning of K-Dependence Bayesian Network Classifier

K-dependence Bayesian network classifier(KDB) has been widely used in data mining and machine learning. To enhance the expression ability and classification performance of KDB, the present study proposes a scalable k-dependence Bayesian classifier (SKDB), which is an extension of the classic KDB algorithm. SKDB introduces a novel attribute sorting method to pre-determine the optimal attribute order and a filtering mechanism to eliminate weak conditional dependence. Experimental results on 30 datasets from the University of California at Irvine (UCI) machine learning repository demonstrate that the above two techniques exert a positive impact on the classification performance of KDB. The proposed algorithm(SKDB) achieves better classification performance compared with several state-of-the-art BNCs (such as Naïve Bayes, tree-augmented Naïve Bayes, averaged one-dependence estimators, and KDB) in terms of 0–1 loss, bias, and variance.

[1]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[2]  R. W. Robinson.  Counting unlabeled acyclic digraphs , 1977 .

[3]  W. Marsden.  I and J , 2012 .

[4]  Zhihua Cai,et al.  Attribute Weighting via Differential Evolution Algorithm for Attribute Weighted Naive Bayes (WNB) , 2011 .

[5]  Liangxiao Jiang,et al.  Improving Tree augmented Naive Bayes for class probability estimation , 2012, Knowl. Based Syst..

[6]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[7]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[8]  Mark A. Hall,et al.  A decision tree-based attribute weighting filter for naive Bayes , 2006, Knowl. Based Syst..

[9]  Michael G. Madden,et al.  On the classification performance of TAN and general Bayesian networks , 2008, Knowl. Based Syst..

[10]  Anderson Ara,et al.  Bagging k-dependence probabilistic networks: An alternative powerful fraud detection tool , 2012, Expert Syst. Appl..

[11]  Davut Hanbay,et al.  E-Mail Classification Using Natural Language Processing , 2019, 2019 27th Signal Processing and Communications Applications Conference (SIU).

[12]  Geoffrey I. Webb,et al.  Scalable Learning of Bayesian Network Classifiers , 2016, J. Mach. Learn. Res..

[13]  Peng Zhang,et al.  SODE: Self-Adaptive One-Dependence Estimators for classification , 2016, Pattern Recognit..

[14]  Hong-Bo Shi,et al.  Tree-augmented naive Bayes ensembles , 2004, Proceedings of 2004 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.04EX826).

[15]  Harry Zhang,et al.  Learning weighted naive Bayes with accurate ranking , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[16]  Liangxiao Jiang,et al.  Learning Tree Augmented Naive Bayes for Ranking , 2005, DASFAA.

[17]  Arcadio Rubio,et al.  Flexible learning of k-dependence Bayesian network classifiers , 2011, GECCO '11.

[18]  Concha Bielza,et al.  Discrete Bayesian Network Classifiers , 2014, ACM Comput. Surv..

[19]  Limin Wang,et al.  General and Local: Averaged k-Dependence Bayesian Classifiers , 2015, Entropy.

[20]  Kirk G. Osadetz,et al.  Geological risk evaluation using the Support Vector Machine with examples from the late Triassic–early Jurassic structural play in western Sverdrup Basin, Canadian Arctic Archipelago , 2012 .

[21]  Franz Pernkopf,et al.  Efficient Heuristics for Discriminative Structure Learning of Bayesian Network Classifiers , 2010, J. Mach. Learn. Res..

[22]  Hongbo Ren,et al.  Spatial Distribution Prediction of Oil and Gas Based on Bayesian Network with Case Study , 2020 .

[23]  John M. Noble,et al.  Bayesian Networks: An Introduction , 2009 .

[24]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[25]  Liangxiao Jiang,et al.  Weightily Averaged One-Dependence Estimators , 2006, PRICAI.

[26]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[27]  Shasha Wang,et al.  Deep feature weighting for naive Bayes and its application to text classification , 2016, Eng. Appl. Artif. Intell..

[28]  Geoffrey I. Webb,et al.  Learning by extrapolation from marginal to full-multivariate probability distributions: decreasingly naive Bayesian classification , 2011, Machine Learning.

[29]  Mehran Sahami,et al.  Learning Limited Dependence Bayesian Classifiers , 1996, KDD.

[30]  Limin Wang,et al.  K-Dependence Bayesian Classifier Ensemble , 2017, Entropy.

[31]  Igor Kononenko,et al.  Machine learning for medical diagnosis: history, state of the art and perspective , 2001, Artif. Intell. Medicine.

[32]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[33]  Pedro M. Domingos,et al.  On the Optimality of the Simple Bayesian Classifier under Zero-One Loss , 1997, Machine Learning.

[34]  Rui Zhang,et al.  Spatial prediction of oil and gas distribution using Tree Augmented Bayesian network , 2020, Comput. Geosci..

[35]  David Maxwell Chickering,et al.  Large-Sample Learning of Bayesian Networks is NP-Hard , 2002, J. Mach. Learn. Res..

[36]  Yang Liu,et al.  Efficient Heuristics for Structure Learning of k-Dependence Bayesian Classifier , 2018, Entropy.

[37]  Abinash Panda,et al.  pgmpy: Probabilistic Graphical Models using Python , 2015, SciPy.

[38]  Eamonn J. Keogh,et al.  Learning augmented Bayesian classifiers: A comparison of distribution-based and classification-based approaches , 1999, AISTATS.

[39]  Yang Liu,et al.  Universal Target Learning: An Efficient and Effective Technique for Semi-Naive Bayesian Learning , 2019, Entropy.

[40]  Geoffrey I. Webb,et al.  Not So Naive Bayes: Aggregating One-Dependence Estimators , 2005, Machine Learning.

[41]  Geoffrey I. Webb,et al.  Selective AnDE for large data learning: a low-bias memory constrained approach , 2017, Knowledge and Information Systems.

[42]  Franz Pernkopf,et al.  Stochastic margin-based structure learning of Bayesian network classifiers , 2013, Pattern Recognit..

[43]  Minghui Sun,et al.  Robust Structure Learning of Bayesian Network by Identifying Significant Dependencies , 2019, IEEE Access.

[44]  Kewei Tu,et al.  Learning Bayesian network structures under incremental construction curricula , 2017, Neurocomputing.

[45]  Rodolfo Ostos,et al.  Machine Learning Approach to Extract Diagnostic and Prognostic Thresholds: Application in Prognosis of Cardiovascular Mortality , 2012, Comput. Math. Methods Medicine.

[46]  Jason D. M. Rennie.  ifile: An Application of Machine Learning to E-Mail Filtering , 2000 .

[47]  Liangxiao Jiang,et al.  A Novel Bayes Model: Hidden Naive Bayes , 2009, IEEE Transactions on Knowledge and Data Engineering.

[48]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .