Knowledge Injection via ML-based Initialization of Neural Networks

Despite the success of artificial neural networks (ANNs) for various complex tasks, their performance and training duration heavily rely on several factors. In many application domains these requirements, such as high data volume and quality, are not satisfied. To tackle this issue, different ways to inject existing domain knowledge into the ANN generation provided promising results. However, the initialization of ANNs is mostly overlooked in this paradigm and remains an important scientific challenge. In this paper, we present a machine learning framework enabling an ANN to perform a semantic mapping from a well-defined, symbolic representation of domain knowledge to weights and biases of an ANN in a specified architecture.

[1]  Nikhil Muralidhar,et al.  Incorporating Prior Domain Knowledge into Deep Neural Networks , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[2]  Junfei Qiao,et al.  Mutual information based weight initialization method for sigmoidal feedforward neural networks , 2016, Neurocomputing.

[3]  Seyyed Ali Seyyedsalehi,et al.  A fast and efficient pre-training method based on layer-by-layer maximum discrimination for deep neural networks , 2015, Neurocomputing.

[4]  Yoshua Bengio,et al.  Exploring Strategies for Training Deep Neural Networks , 2009, J. Mach. Learn. Res..

[5]  Jude W. Shavlik,et al.  Knowledge-Based Artificial Neural Networks , 1994, Artif. Intell..

[6]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[7]  Nagiza F. Samatova,et al.  Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data , 2016, IEEE Transactions on Knowledge and Data Engineering.

[8]  E. Fiesler,et al.  Neural Network Initialization , 1995, IWANN.

[9]  J. L. Peterson,et al.  Deep Neural Network Initialization With Decision Trees , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Randall Balestriero,et al.  Neural Decision Trees , 2017, ArXiv.

[12]  Charu C. Aggarwal,et al.  Using a Random Forest to Inspire a Neural Network and Improving on It , 2017, SDM.

[13]  Andrea Borghesi,et al.  Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey , 2020, ArXiv.

[14]  Wee Kheng Leow,et al.  On mapping decision trees and neural networks , 1999, Knowl. Based Syst..

[15]  Wei Lu,et al.  Integrating Machine Learning with Human Knowledge , 2020, iScience.

[16]  Hisashi Shimodaira,et al.  A weight value initialization method for improving learning performance of the backpropagation algorithm in neural networks , 1994, Proceedings Sixth International Conference on Tools with Artificial Intelligence. TAI 94.

[17]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[18]  Yoshua Bengio,et al.  Knowledge Matters: Importance of Prior Information for Optimization , 2013, J. Mach. Learn. Res..

[19]  Eric Xing,et al.  Deep Generative Models with Learnable Knowledge Constraints , 2018, NeurIPS.

[20]  Marco Gori,et al.  Integrating Prior Knowledge into Deep Learning , 2017, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[21]  Hussein Alnuweiri,et al.  Acceleration of back propagation through initial weight pre-training with delta rule , 1993, IEEE International Conference on Neural Networks.

[22]  Pascal Larzabal,et al.  Initialization of Supervised Training for Parametric Estimation , 1999, Neural Processing Letters.

[23]  Guy Van den Broeck,et al.  A Semantic Loss Function for Deep Learning with Symbolic Knowledge , 2017, ICML.

[24]  Arunava Banerjee.  Initializing neural networks using decision trees , 1997, COLT 1997.

[25]  Erwan Scornet,et al.  Neural Random Forests , 2016, Sankhya A.

[26]  Rahul Rai,et al.  Driven by Data or Derived Through Physics? A Review of Hybrid Physics Guided Machine Learning Techniques With Cyber-Physical System (CPS) Focus , 2020, IEEE Access.

[27]  Victor S. Sheng,et al.  A Roadmap to Domain Knowledge Integration in Machine Learning , 2020, 2020 IEEE International Conference on Knowledge Graph (ICKG).

[28]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[29]  Jiri Matas,et al.  All you need is a good init , 2015, ICLR.

[30]  Miroslav Kubat,et al.  Initialization of neural networks by means of decision trees , 1995, Knowl. Based Syst..

[31]  Olac Fuentes,et al.  Improving Weight Initialization of ReLU and Output Layers , 2019, ICANN.

[32]  Birgit Kirsch,et al.  Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems , 2019 .

[33]  Ashwin Srinivasan,et al.  Incorporating Domain Knowledge into Deep Neural Networks , 2021, ArXiv.