Class-specific attribute value weighting for Naive Bayes

Abstract Naive Bayes (NB) is one of the top 10 data mining algorithms. However, its assumption of conditional independence rarely holds true in real-world applications. To alleviate this assumption, numerous attribute weighting approaches have been proposed. However, few of these simultaneously pay attention to the horizontal granularity of attribute values and vertical granularity of class labels. In this study, we propose a new paradigm for fine-grained attribute weighting, named class-specific attribute value weighting. For each class, this approach discriminatively assigns a specific weight to each attribute value. We refer to the resulting improved model as class-specific attribute value weighted NB (CAVWNB). In CAVWNB, the class-specific attribute value weight matrix is learned by either maximizing the conditional log-likelihood (CLL) or minimizing the mean squared error (MSE). Thus, two versions are proposed, which we denote as CAVWNBCLL and CAVWNBMSE, respectively. Extensive experimental results on a large number of datasets show that both CAVWNBCLL and CAVWNBMSE significantly outperform NB and all the other existing state-of-the-art attribute weighting approaches used for comparison.

[1]  Zhiqiang Geng,et al.  A new deep belief network based on RBM with glial chains , 2018, Inf. Sci..

[2]  Khalil el Hindi,et al.  Selectively Fine-Tuning Bayesian Network Learning Algorithm , 2016, Int. J. Pattern Recognit. Artif. Intell..

[3]  Shasha Wang,et al.  Deep feature weighting for naive Bayes and its application to text classification , 2016, Eng. Appl. Artif. Intell..

[4]  Shasha Wang,et al.  Cost-sensitive Bayesian network classifiers , 2014, Pattern Recognit. Lett..

[5]  Shasha Wang,et al.  Structure extended multinomial naive Bayes , 2016, Inf. Sci..

[6]  Jia Wu,et al.  A Correlation-Based Feature Weighting Filter for Naive Bayes , 2019, IEEE Transactions on Knowledge and Data Engineering.

[7]  Geoffrey I. Webb,et al.  Alleviating naive Bayes attribute independence assumption by attribute weighting , 2013, J. Mach. Learn. Res..

[8]  Liangxiao Jiang,et al.  Class-specific attribute weighted naive Bayes , 2019, Pattern Recognit..

[9]  Pat Langley,et al.  An Analysis of Bayesian Classifiers , 1992, AAAI.

[10]  Paul H. J. Kelly,et al.  Performance prediction of paging workloads using lightweight tracing , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[11]  Jose Miguel Puerta,et al.  Speeding up incremental wrapper feature subset selection with Naive Bayes classifier , 2014, Knowl. Based Syst..

[12]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[13]  Qin Wei,et al.  A model-free Bayesian classifier , 2019, Inf. Sci..

[14]  Liangxiao Jiang,et al.  A Novel Bayes Model: Hidden Naive Bayes , 2009, IEEE Transactions on Knowledge and Data Engineering.

[15]  Chang-Hwan Lee.  A gradient approach for value weighted classification learning in naive Bayes , 2015, Knowl. Based Syst..

[16]  Chang-Hwan Lee.  An information-theoretic filter approach for value weighted classification learning in naive Bayes , 2018, Data Knowl. Eng..

[17]  Bo Tang,et al.  Toward Optimal Feature Selection in Naive Bayes for Text Categorization , 2016, IEEE Transactions on Knowledge and Data Engineering.

[18]  Liangxiao Jiang,et al.  Not so greedy: Randomly Selected Naive Bayes , 2012, Expert Syst. Appl..

[19]  Marcin Korytkowski,et al.  Fast image classification by boosting fuzzy classifiers , 2016, Inf. Sci..

[20]  Liangxiao Jiang,et al.  Toward naive Bayes with attribute value weighting , 2019, Neural Computing and Applications.

[21]  Dejing Dou,et al.  Calculating Feature Weights in Naive Bayes with Kullback-Leibler Measure , 2011, 2011 IEEE 11th International Conference on Data Mining.

[22]  Peng Zhang,et al.  SODE: Self-Adaptive One-Dependence Estimators for classification , 2016, Pattern Recognit..

[23]  Bernhard Pfahringer,et al.  Locally Weighted Naive Bayes , 2002, UAI.

[24]  Harry Zhang,et al.  Learning weighted naive Bayes with accurate ranking , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[25]  Philip S. Yu,et al.  Top 10 algorithms in data mining , 2007, Knowledge and Information Systems.

[26]  Shasha Wang,et al.  Adapting naive Bayes tree for text classification , 2015, Knowledge and Information Systems.

[27]  Ling Wang,et al.  Finding Multiple Roots of Nonlinear Equation Systems via a Repulsion-Based Adaptive Differential Evolution , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[28]  Pedro M. Domingos,et al.  Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier , 1996, ICML.

[29]  Jesús Alcalá-Fdez,et al.  KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework , 2011, J. Multiple Valued Log. Soft Comput..

[30]  Liangxiao Jiang,et al.  Learning decision tree for ranking , 2009, Knowledge and Information Systems.

[31]  Wenyin Gong,et al.  Enhancing the performance of differential evolution using orthogonal design method , 2008, Appl. Math. Comput..

[32]  George Forman,et al.  An Extensive Empirical Study of Feature Selection Metrics for Text Classification , 2003, J. Mach. Learn. Res..

[33]  Jia Wu,et al.  Artificial immune system for attribute weighted Naive Bayes classification , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[34]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[35]  Jia Wu,et al.  Self-adaptive probability estimation for Naive Bayes classification , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[36]  Khalil el Hindi,et al.  Fine tuning the Naïve Bayesian learning algorithm , 2014, AI Commun..

[37]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[38]  Liangxiao Jiang,et al.  Bayesian Citation-KNN with distance weighting , 2014, Int. J. Mach. Learn. Cybern..

[39]  Liangxiao Jiang,et al.  Attribute Value Weighted Average of One-Dependence Estimators , 2017, Entropy.

[40]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[41]  Liangxiao Jiang,et al.  An attribute value frequency-based instance weighting filter for naive Bayes , 2018, J. Exp. Theor. Artif. Intell..

[42]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[43]  Diab M. Diab,et al.  Using differential evolution for fine tuning naïve Bayesian classifiers and its application for text classification , 2017, Appl. Soft Comput..

[44]  Hongwei Li,et al.  Local value difference metric , 2014, Pattern Recognit. Lett..

[45]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[46]  Elena Marchiori,et al.  Class Dependent Feature Weighting and K-Nearest Neighbor Classification , 2013, PRIB.

[47]  Wenyin Gong,et al.  Parameter extraction of solar cell models using repaired adaptive differential evolution , 2013 .