Feature Importance Measurement based on Decision Tree Sampling

Random forest is effective for prediction tasks but the randomness of tree generation hinders interpretability in feature importance analysis. To address this, we proposed DT-Sampler, a SAT-based method for measuring feature importance in tree-based model. Our method has fewer parameters than random forest and provides higher interpretability and stability for the analysis in real-world problems. An implementation of DT-Sampler is available at https://github.com/tsudalab/DT-sampler.

[1]  C. Rudin,et al.  FasterRisk: Fast and Accurate Interpretable Risk Scores , 2022, NeurIPS.

[2]  I. Takeuchi,et al.  A Confidence Machine for Sparse High-Order Interaction Model , 2022, ArXiv.

[3]  Ichiro Takeuchi,et al.  Fast and More Powerful Selective Inference for Sparse High-order Interaction Model , 2021, AAAI.

[4]  Cesare Tinelli,et al.  Handbook of Satisfiability , 2021, Handbook of Satisfiability.

[5]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[6]  Siegfried Nijssen,et al.  Learning optimal decision trees using constraint programming , 2020, Constraints.

[7]  Mikoláš Janota,et al.  SAT-Based Encodings for Optimal Decision Trees with Explicit Paths , 2020, SAT.

[8]  Mate Soos,et al.  Tinted, Detached, and Lazy CNF-XOR Solving and Its Applications to Counting and Sampling , 2020, CAV.

[9]  C. Rudin,et al.  In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction , 2020, Journal of Quantitative Criminology.

[10]  Amina Adadi,et al.  Explainable AI for Healthcare: From Black Box to Interpretable Models , 2020 .

[11]  Koji Tsuda,et al.  An interpretable machine learning model for diagnosis of Alzheimer's disease , 2019, PeerJ.

[12]  Cynthia Rudin,et al.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , 2018, Nature Machine Intelligence.

[13]  Joao Marques-Silva,et al.  Learning Optimal Decision Trees with SAT , 2018, IJCAI.

[14]  Ankur Teredesai,et al.  Interpretable Machine Learning in Healthcare , 2018, 2018 IEEE International Conference on Healthcare Informatics (ICHI).

[15]  Koushik Sen,et al.  Efficient Sampling of SAT Solutions for Testing , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[16]  Shamim Nemati,et al.  An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU , 2017, Critical care medicine.

[17]  Margo I. Seltzer,et al.  Learning Certifiably Optimal Rule Lists , 2017, KDD.

[18]  Jure Leskovec,et al.  Interpretable Decision Sets: A Joint Framework for Description and Prediction , 2016, KDD.

[19]  Seth Flaxman,et al.  European Union Regulations on Algorithmic Decision-Making and a "Right to Explanation" , 2016, AI Mag..

[20]  Cynthia Rudin,et al.  Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model , 2015, ArXiv.

[21]  Barry O'Sullivan,et al.  Minimising Decision Tree Size as Combinatorial Optimisation , 2009, CP.

[22]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[23]  Jianqing Fan,et al.  Sure independence screening for ultrahigh dimensional feature space , 2006, math/0612857.

[24]  Edward I. George,et al.  Bayesian Treed Models , 2002, Machine Learning.

[25]  L. Breiman.  Random Forests , 2001, Encyclopedia of Machine Learning and Data Mining.

[26]  H. Chipman,et al.  Bayesian CART Model Search , 1998 .

[27]  Adrian F. M. Smith,et al.  A Bayesian CART algorithm , 1998 .

[28]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[29]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[30]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[31]  K. Tsuda,et al.  Self-learning Entropic Population Annealing for Interpretable Materials Design , 2022, Digital Discovery.

[32]  H. Chipman,et al.  Bayesian Additive Regression Trees , 2006 .

[33]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[34]  G. S. Tseitin.  On the Complexity of Derivation in Propositional Calculus , 1983 .