How Does Knowledge Injection Help in Informed Machine Learning?

Informed machine learning describes the injection of prior knowledge into learning systems. It can help to improve generalization, especially when training data is scarce. However, the field is so application-driven that general analyses about the effect of knowledge injection are rare. This makes it difficult to transfer existing approaches to new applications, or to estimate potential improvements. Therefore, in this paper, we present a framework for quantifying the value of prior knowledge in informed machine learning. Our main contributions are threefold. Firstly, we propose a set of relevant metrics for quantifying the benefits of knowledge injection, comprising in-distribution accuracy, out-of-distribution robustness, and knowledge conformity. We also introduce a metric that combines performance improvement and data reduction. Secondly, we present a theoretical framework that represents prior knowledge in a function space and relates it to data representations and a trained model. This suggests that the distances between knowledge and data influence potential model improvements. Thirdly, we perform a systematic experimental study with controllable toy problems. All in all, this helps to find general answers to the question how knowledge injection helps in informed machine learning.

[1]  G. Malnati,et al.  Theory-Guided Deep Learning Algorithms: An Experimental Evaluation , 2022, Electronics.

[2]  Jianyi Yang,et al.  Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity , 2022, ICML.

[3]  Laura von Rueden,et al.  Informed Pre-Training on Prior Knowledge , 2022, ArXiv.

[4]  Syed Tahseen Raza Rizvi,et al.  Knowledge Augmented Machine Learning with Applications in Autonomous Driving: A Survey , 2022, ArXiv.

[5]  Md Kamruzzaman Sarker,et al.  Neuro-symbolic artificial intelligence , 2021, AI Commun..

[6]  Peng Cui,et al.  Towards Out-Of-Distribution Generalization: A Survey , 2021, ArXiv.

[7]  Nezihe Merve Gurel,et al.  Knowledge Enhanced Machine Learning Pipeline against Diverse Adversarial Attacks , 2021, ICML.

[8]  I. Kevrekidis,et al.  Physics-informed machine learning , 2021, Nature Reviews Physics.

[9]  Sebastian Houben,et al.  Explainable Machine Learning with Prior Knowledge: An Overview , 2021, ArXiv.

[10]  Alexandre Alahi,et al.  Injecting Knowledge in Data-driven Vehicle Trajectory Predictors , 2021, Transportation Research Part C: Emerging Technologies.

[11]  Martín Arjovsky.  Out of Distribution Generalization in Machine Learning , 2021, ArXiv.

[12]  Christian Bauckhage,et al.  Street-Map Based Validation of Semantic Segmentation in Autonomous Driving , 2021, 2020 25th International Conference on Pattern Recognition (ICPR).

[13]  Shaolei Ren,et al.  A Quantitative Perspective on Values of Domain Knowledge for Machine Learning , 2020, ArXiv.

[14]  Kush R. Varshney,et al.  Empirical or Invariant Risk Minimization? A Sample Complexity Perspective , 2020, ICLR.

[15]  Rafet Sifa,et al.  Combining Machine Learning and Simulation to a Hybrid Modelling Approach: Current and Future Directions , 2020, IDA.

[16]  G. Karniadakis,et al.  On the Convergence of Physics Informed Neural Networks for Linear Second-Order Elliptic and Parabolic Type PDEs , 2020, Communications in Computational Physics.

[17]  Gary Marcus,et al.  The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence , 2020, ArXiv.

[18]  Mihaela van der Schaar,et al.  Improving Model Robustness Using Causal Knowledge , 2019, ArXiv.

[19]  David Lopez-Paz,et al.  Invariant Risk Minimization , 2019, ArXiv.

[20]  Laura von Rueden,et al.  Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems , 2019, IEEE Transactions on Knowledge and Data Engineering.

[21]  Christian Bauckhage,et al.  Informed Machine Learning - Towards a Taxonomy of Explicit Integration of Knowledge into Machine Learning , 2019, ArXiv.

[22]  V. Vapnik,et al.  Rethinking statistical learning theory: learning using statistical invariants , 2018, Machine Learning.

[23]  Franziska Meier,et al.  Using Simulation to Improve Sample-Efficiency of Bayesian Optimization for Bipedal Robots , 2018, J. Mach. Learn. Res..

[24]  Zhaoqi Wang,et al.  Simulation-assisted machine learning , 2018, Bioinform..

[25]  Anuj Karpatne,et al.  Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling , 2017, ArXiv.

[26]  Nagiza F. Samatova,et al.  Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data , 2016, IEEE Transactions on Knowledge and Data Engineering.

[27]  Stefano Ermon,et al.  Label-Free Supervision of Neural Networks with Physics and Domain Knowledge , 2016, AAAI.

[28]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[29]  Bernhard Schölkopf,et al.  Statistical Learning Theory: Models, Concepts, and Results , 2008, Inductive Logic.

[30]  Vladimir Vapnik,et al.  Principles of Risk Minimization for Learning Theory , 1991, NIPS.

[31]  T. Kyono.  Towards Causally-Aware Machine Learning , 2021 .

[32]  I. Roterman,et al.  Summary , 2020, From Globular Proteins to Amyloids.