SaDe: Learning Models that Provably Satisfy Domain Constraints

With increasing real world applications of machine learning, models are often required to comply with certain domain based requirements, e.g., safety guarantees in aircraft systems, legal constraints in a loan approval model. A natural way to represent these properties is in the form of constraints. Including such constraints in machine learning is typically done by the means of regularization, which does not guarantee satisfaction of the constraints. In this paper, we present a machine learning approach that can handle a wide variety of constraints, and guarantee that these constraints will be satisfied by the model even on unseen data. We cast machine learning as a maximum satisfiability problem, and solve it using a novel algorithm SaDe which combines constraint satisfaction with gradient descent. We demonstrate on three use cases that this approach learns models that provably satisfy the given constraints.

[1]  Eric P. Xing,et al.  Harnessing Deep Neural Networks with Logic Rules , 2016, ACL.

[2]  Andrea Passerini,et al.  Structured learning modulo theories , 2014, Artif. Intell..

[3]  Dmitry Malioutov,et al.  MLIC: A MaxSAT-Based Framework for Learning Interpretable Classification Rules , 2018, CP.

[4]  Yang Li,et al.  Robustness Verification of Tree-based Models , 2019, NeurIPS.

[5]  Guy Van den Broeck,et al.  Counterexample-Guided Learning of Monotonic Neural Networks , 2020, NeurIPS.

[6]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[7]  Mislav Balunovic,et al.  DL2: Training and Querying Neural Networks with Logic , 2019, ICML.

[8]  Timon Gehr,et al.  Boosting Robustness Certification of Neural Networks , 2018, ICLR.

[9]  Roberto Sebastiani,et al.  OptiMathSAT: A Tool for Optimization Modulo Theories , 2015, Journal of Automated Reasoning.

[10]  Toniann Pitassi,et al.  Fairness through awareness , 2011, ITCS '12.

[11]  James Cussens,et al.  Bayesian network learning by compiling to weighted MAX-SAT , 2008, UAI.

[12]  Marie-José Huguet,et al.  Learning Optimal Decision Trees with MaxSAT and its Integration in AdaBoost , 2020, IJCAI.

[13]  Guy Van den Broeck,et al.  A Semantic Loss Function for Deep Learning with Symbolic Knowledge , 2017, ICML.

[14]  Luc De Raedt,et al.  DeepProbLog: Neural Probabilistic Logic Programming , 2018, BNAIC/BENELEARN.

[15]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[16]  Ben Taskar,et al.  Posterior Regularization for Structured Latent Variable Models , 2010, J. Mach. Learn. Res..

[17]  Siegfried Nijssen,et al.  Learning optimal decision trees using constraint programming , 2020, Constraints.

[18]  Nikhil Muralidhar,et al.  Incorporating Prior Domain Knowledge into Deep Neural Networks , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[19]  Yingqian Zhang,et al.  Learning Optimal Classification Trees Using a Binary Linear Program Formulation , 2019, BNAIC/BENELEARN.

[20]  Mykel J. Kochenderfer,et al.  The Marabou Framework for Verification and Analysis of Deep Neural Networks , 2019, CAV.

[21]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[22]  Nikolaj Bjørner,et al.  νZ - An Optimizing SMT Solver , 2015, TACAS.

[23]  Dan Roth,et al.  Semantic Role Labeling Via Integer Linear Programming Inference , 2004, COLING.

[24]  Marco Gori,et al.  Machine Learning: A Constraint-Based Approach , 2017 .

[25]  Marco Gori,et al.  Semantic-based regularization for learning and inference , 2017, Artif. Intell..

[26]  Maurice Bruynooghe,et al.  Predicate logic as a modeling language: modeling and solving some machine learning and data mining problems with IDP3 , 2013, Theory and Practice of Logic Programming.

[27]  Toby Walsh,et al.  Handbook of Constraint Programming , 2006, Handbook of Constraint Programming.

[28]  Matti Järvisalo,et al.  Applications of MaxSAT in Data Analysis , 2019, POS@SAT.

[29]  Yaniv Sa'ar,et al.  Verifying Robustness of Gradient Boosted Models , 2019, AAAI.

[30]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[31]  Stefano Ermon,et al.  Label-Free Supervision of Neural Networks with Physics and Domain Knowledge , 2016, AAAI.

[32]  Luca Oneto,et al.  Fairness in Machine Learning , 2020, INNSBDDL.

[33]  Maria Luisa Bonet,et al.  SAT-based MaxSAT algorithms , 2013, Artif. Intell..