A Unified Framework for Knowledge Intensive Gradient Boosting: Leveraging Human Experts for Noisy Sparse Domains

Incorporating richer human inputs including qualitative constraints such as monotonic and synergistic influences has long been adapted inside AI. Inspired by this, we consider the problem of using such influence statements in the successful gradient-boosting framework. We develop a unified framework for both classification and regression settings that can both effectively and efficiently incorporate such constraints to accelerate learning to a better model. Our results in a large number of standard domains and two particularly novel real-world domains demonstrate the superiority of using domain knowledge rather than treating the human as a mere labeler.

[1]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[2]  William Marsh,et al.  Combining data and meta-analysis to build Bayesian networks for clinical decision support , 2014, J. Biomed. Informatics.

[3]  Toshihide Ibaraki,et al.  Data Analysis by Positive Decision Trees , 1999, CODAS.

[4]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[5]  A. J. Feelders,et al.  Classification trees for problems with monotonicity constraints , 2002, SKDD.

[6]  Pedro Antonio Gutiérrez,et al.  Monotonic classification: an overview on algorithms, performance measures and data sets , 2018, Neurocomputing.

[7]  Maya R. Gupta,et al.  Deep Lattice Networks and Partial Monotonic Functions , 2017, NIPS.

[8]  Glenn Fung,et al.  Knowledge-Based Support Vector Machine Classifiers , 2002, NIPS.

[9]  Michael P. Wellman.  Fundamental Concepts of Qualitative Probabilistic Networks , 1990, Artif. Intell..

[10]  Qiang Ji,et al.  Constrained Maximum Likelihood Learning of Bayesian Networks for Facial Action Recognition , 2008, ECCV.

[11]  Yoshua Bengio,et al.  Tackling Climate Change with Machine Learning , 2019, ACM Comput. Surv..

[12]  Gautam Kunapuli,et al.  The Adviceptron: Giving Advice to the Perceptron , 2010 .

[13]  A. J. Feelders,et al.  Pruning for Monotone Classification Trees , 2003, IDA.

[14]  Sriraam Natarajan,et al.  Guiding Autonomous Agents to Better Behaviors through Human Advice , 2013, 2013 IEEE 13th International Conference on Data Mining.

[15]  Wei Liu,et al.  Enhanced Random Forest Algorithms for Partially Monotone Ordinal Classification , 2019, AAAI.

[16]  Thomas G. Dietterich,et al.  Learning from Sparse Data by Exploiting Monotonicity Constraints , 2005, UAI.

[17]  Shuo Yang,et al.  Knowledge Intensive Learning: Combining Qualitative Constraints with Causal Independence for Parameter Learning in Probabilistic Models , 2013, ECML/PKDD.

[18]  A. J. Feelders,et al.  Isotonic Classification Trees , 2009, IDA.

[19]  Francisco Herrera,et al.  Monotonic Random Forest with an Ensemble Pruning Mechanism based on the Degree of Monotonicity , 2015, New Generation Computing.

[20]  John D. Lafferty,et al.  Prediction Rule Reshaping , 2018, ICML.

[21]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[22]  A. J. Feelders,et al.  Nearest Neighbour Classification with Monotonicity Constraints , 2008, ECML/PKDD.

[23]  Daniel B. Neill,et al.  Machine Learning for the Developing World , 2018, ACM Trans. Manag. Inf. Syst..

[24]  Sriraam Natarajan,et al.  Human-Guided Learning for Probabilistic Logic Models , 2018, Front. Robot. AI.

[25]  Kristian Kersting,et al.  Learning from Imbalanced Data in Relational Domains: A Soft Margin Approach , 2014, 2014 IEEE International Conference on Data Mining.

[26]  Ingoo Han,et al.  The discovery of experts' decision rules from qualitative bankruptcy data using genetic algorithms , 2003, Expert Syst. Appl..

[27]  Qiang Ji,et al.  Learning Bayesian Networks with qualitative constraints , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Jude W. Shavlik,et al.  Knowledge-Based Artificial Neural Networks , 1994, Artif. Intell..

[29]  Mark Reynolds,et al.  Effective Monotone Knowledge Integration in Kernel Support Vector Machines , 2016, ADMA.

[30]  Francisco Herrera,et al.  Managing Monotonicity in Classification by a Pruned AdaBoost , 2016, HAIS.

[31]  J. Bioch,et al.  Monotone Decision Trees and Noisy Data , 2002 .

[32]  Chih-Chuan Chen,et al.  Credit rating with a monotonicity-constrained support vector machine model , 2014, Expert Syst. Appl..

[33]  Tie-Yan Liu,et al.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree , 2017, NIPS.

[34]  A. Ben-David.  Monotonicity Maintenance in Information-Theoretic Machine Learning Algorithms , 1995, Machine Learning.

[35]  F. T. Wright,et al.  Order restricted statistical inference , 1988 .

[36]  P. Anglin,et al.  SEMIPARAMETRIC ESTIMATION OF A HEDONIC PRICE FUNCTION , 1996 .

[37]  Jan C. Bioch,et al.  A Decision Tree Algorithm for Ordinal Classification , 1999, IDA.