A novel selective naïve Bayes algorithm

Abstract Naive Bayes is one of the most popular data mining algorithms. Its efficiency comes from the assumption of attribute independence, although this might be violated in many real-world data sets. Many efforts have been done to mitigate the assumption, among which attribute selection is an important approach. However, conventional efforts to perform attribute selection in naive Bayes suffer from heavy computational overhead. This paper proposes an efficient selective naive Bayes algorithm, which adopts only some of the attributes to construct selective naive Bayes models. These models are built in such a way that each one is a trivial extension of another. The most predictive selective naive Bayes model can be selected by the measures of incremental leave-one-out cross validation. As a result, attributes can be selected by efficient model selection. Empirical results demonstrate that the selective naive Bayes shows superior classification accuracy, yet at the same time maintains the simplicity and efficiency.

[1]  Liangxiao Jiang,et al.  Attribute Value Weighted Average of One-Dependence Estimators , 2017, Entropy.

[2]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[3]  Limin Wang,et al.  Combining decision tree and Naive Bayes for classification , 2006, Knowl. Based Syst..

[4]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[5]  Liangxiao Jiang,et al.  Not so greedy: Randomly Selected Naive Bayes , 2012, Expert Syst. Appl..

[6]  Geoffrey I. Webb,et al.  Learning by extrapolation from marginal to full-multivariate probability distributions: decreasingly naive Bayesian classification , 2011, Machine Learning.

[7]  Dejing Dou,et al.  Calculating Feature Weights in Naive Bayes with Kullback-Leibler Measure , 2011, 2011 IEEE 11th International Conference on Data Mining.

[8]  Liangxiao Jiang,et al.  Not always simple classification: Learning SuperParent for class probability estimation , 2015, Expert Syst. Appl..

[9]  Irina Rish,et al.  An empirical study of the naive Bayes classifier , 2001 .

[10]  Geoffrey I. Webb,et al.  A Comparative Study of Bandwidth Choice in Kernel Density Estimation for Naive Bayesian Classification , 2009, PAKDD.

[11]  Mark A. Hall,et al.  A decision tree-based attribute weighting filter for naive Bayes , 2006, Knowl. Based Syst..

[12]  Remco R. Bouckaert.  Naive Bayes Classifiers That Perform Well with Continuous Variables , 2004, Australian Conference on Artificial Intelligence.

[13]  Liangxiao Jiang,et al.  A Novel Bayes Model: Hidden Naive Bayes , 2009, IEEE Transactions on Knowledge and Data Engineering.

[14]  Liangxiao Jiang,et al.  Random one-dependence estimators , 2011, Pattern Recognit. Lett..

[15]  José A. Gámez,et al.  Domains of competence of the semi-naive Bayesian network classifiers , 2014, Inf. Sci..

[16]  Mark A. Hall,et al.  Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning , 1999, ICML.

[17]  Yu-Lin He,et al.  Non-Naive Bayesian Classifiers for Classification Problems With Continuous Attributes , 2014, IEEE Transactions on Cybernetics.

[18]  Geoffrey I. Webb,et al.  Not So Naive Bayes: Aggregating One-Dependence Estimators , 2005, Machine Learning.

[19]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[20]  Geoffrey I. Webb,et al.  Selective AnDE for large data learning: a low-bias memory constrained approach , 2017, Knowledge and Information Systems.

[21]  Mehran Sahami,et al.  Learning Limited Dependence Bayesian Classifiers , 1996, KDD.

[22]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[23]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[24]  Geoffrey I. Webb,et al.  Alleviating naive Bayes attribute independence assumption by attribute weighting , 2013, J. Mach. Learn. Res..

[25]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[26]  Sasan H. Alizadeh,et al.  Mixture of latent multinomial naive Bayes classifier , 2018, Appl. Soft Comput..

[27]  Liangxiao Jiang,et al.  Two feature weighting approaches for naive Bayes text classifiers , 2016, Knowl. Based Syst..

[28]  Huan Liu,et al.  Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution , 2003, ICML.

[29]  Liangxiao Jiang,et al.  Hidden Naive Bayes , 2005, AAAI.

[30]  Chang-Hwan Lee.  A gradient approach for value weighted classification learning in naive Bayes , 2015, Knowl. Based Syst..

[31]  Eamonn J. Keogh,et al.  Learning augmented Bayesian classifiers: A comparison of distribution-based and classification-based approaches , 1999, AISTATS.

[32]  Thomas D. Nielsen,et al.  Classification using Hierarchical Naïve Bayes models , 2006, Machine Learning.

[33]  Chang-Hwan Lee.  An information-theoretic filter approach for value weighted classification learning in naive Bayes , 2018, Data Knowl. Eng..

[34]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[35]  Jose Miguel Puerta,et al.  Speeding up incremental wrapper feature subset selection with Naive Bayes classifier , 2014, Knowl. Based Syst..

[36]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[37]  Harry Zhang,et al.  Learning weighted naive Bayes with accurate ranking , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[38]  Philip S. Yu,et al.  Top 10 algorithms in data mining , 2007, Knowledge and Information Systems.

[39]  Ron Kohavi,et al.  The Power of Decision Tables , 1995, ECML.

[40]  Jia Wu,et al.  A Correlation-Based Feature Weighting Filter for Naive Bayes , 2019, IEEE Transactions on Knowledge and Data Engineering.

[41]  Bojan Cestnik,et al.  Estimating Probabilities: A Crucial Task in Machine Learning , 1990, ECAI.

[42]  Pat Langley,et al.  Induction of Selective Bayesian Classifiers , 1994, UAI.

[43]  Jonathan M. Garibaldi,et al.  A 'non-parametric' version of the naive Bayes classifier , 2011, Knowl. Based Syst..