Knowledge-based Residual Learning

Small data has been a barrier for many machine learning tasks, especially when applied in scientific domains. Fortunately, we can utilize domain knowledge to make up the lack of data. Hence, in this paper, we propose a hybrid model KRL that treats domain knowledge model as a weak learner and uses another neural net model to boost it. We prove that KRL is guaranteed to improve over pure domain knowledge model and pure neural net model under certain loss functions. Extensive experiments have shown the superior performance of KRL over baselines. In addition, several case studies have explained how the domain knowledge can assist the prediction.

[1]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[2]  Nitesh V. Chawla,et al.  SMOTEBoost: Improving Prediction of the Minority Class in Boosting , 2003, PKDD.

[3]  William L. Jorgensen,et al.  Journal of Chemical Information and Modeling , 2005, J. Chem. Inf. Model..

[4]  T. Michael Knasel,et al.  Robotics and autonomous systems , 1988, Robotics Auton. Syst..

[5]  Anuj Karpatne,et al.  Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling , 2017, ArXiv.

[6]  Guanjie Zheng,et al.  Learning to Route via Theory-Guided Residual Network , 2021, ArXiv.

[7]  Jinlong Wu,et al.  Physics-informed machine learning approach for reconstructing Reynolds stress modeling discrepancies based on DNS data , 2016, 1606.07987.

[8]  Lin Zhang,et al.  Inferring fine-grained air pollution map via a spatiotemporal super-resolution scheme , 2019, UbiComp/ISWC Adjunct.

[9]  Ohad Shamir,et al.  Are ResNets Provably Better than Linear Predictors? , 2018, NeurIPS.

[10]  John Langford,et al.  Learning Deep ResNet Blocks Sequentially using Boosting Theory , 2017, ICML.

[11]  T. Wigley,et al.  Statistical downscaling of general circulation model output: A comparison of methods , 1998 .

[12]  Xi Chen,et al.  Global Monitoring of Inland Water Dynamics: State-of-the-Art, Challenges, and Opportunities , 2016, Computational Sustainability.

[13]  João Gama,et al.  Predicting Taxi–Passenger Demand Using Streaming Data , 2013, IEEE Transactions on Intelligent Transportation Systems.

[14]  Anuj Karpatne,et al.  BHPMF – a hierarchical Bayesian approach to gap-filling and trait prediction for macroecology and functional biogeography , 2015 .

[15]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[16]  Jason Yosinski,et al.  Hamiltonian Neural Networks , 2019, NeurIPS.

[17]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[18]  Lihong Li,et al.  An Empirical Evaluation of Thompson Sampling , 2011, NIPS.

[19]  Benjamin W. Wah,et al.  Editorial: Two Named to Editorial Board of IEEE Transactions on Knowledge and Data Engineering , 1996 .

[20]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Anuj Karpatne,et al.  Physics Guided RNNs for Modeling Dynamical Systems: A Case Study in Simulating Lake Temperature Profiles , 2018, SDM.

[22]  Jiajun Wu,et al.  Combining Physical Simulators and Object-Based Networks for Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[23]  Current Biology , 2012, Current Biology.

[24]  Zhe Jiang,et al.  Monitoring Land-Cover Changes: A Machine-Learning Perspective , 2016, IEEE Geoscience and Remote Sensing Magazine.

[25]  Philip S. Yu,et al.  2014 IEEE International Conference on Data Mining , 2014 .

[26]  Nagiza F. Samatova,et al.  Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data , 2016, IEEE Transactions on Knowledge and Data Engineering.