STLnet: Signal Temporal Logic Enforced Multivariate Recurrent Neural Networks

Recurrent Neural Networks (RNNs) have made great achievements for sequential prediction tasks. In practice, the target sequence often follows certain model properties or patterns (e.g., reasonable ranges, consecutive changes, resource constraint, temporal correlations between multiple variables, existence, unusual cases, etc.). However, RNNs cannot guarantee their learned distributions satisfy these properties. It is even more challenging for the prediction of large-scale and complex Cyber-Physical Systems. Failure to produce outcomes that meet these properties will result in inaccurate and even meaningless results. In this paper, we develop a new temporal logic-based learning framework, STLnet, which guides the RNN learning process with auxiliary knowledge of model properties, and produces a more robust model for improved future predictions. Our framework can be applied to general sequential deep learning models, and trained in an endto-end manner with back-propagation. We evaluate the performance of STLnet using large-scale real-world city data. The experimental results show STLnet not only improves the accuracy of predictions, but importantly also guarantees the satisfaction of model properties and increases the robustness of RNNs.

[1]  Stefano Ermon,et al.  Label-Free Supervision of Neural Networks with Physics and Domain Knowledge , 2016, AAAI.

[2]  Lu Feng,et al.  SaSTL: Spatial Aggregation Signal Temporal Logic for Runtime Monitoring in Smart Cities , 2019, 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS).

[3]  Dejan Nickovic,et al.  Specification-Based Monitoring of Cyber-Physical Systems: A Survey on Theory, Tools and Applications , 2018, Lectures on Runtime Verification.

[4]  Eric P. Xing,et al.  Toward Controlled Generation of Text , 2017, ICML.

[5]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[6]  Alberto L. Sangiovanni-Vincentelli,et al.  Model predictive control with signal temporal logic specifications , 2014, 53rd IEEE Conference on Decision and Control.

[7]  Jude Shavlik,et al.  Refinement ofApproximate Domain Theories by Knowledge-Based Neural Networks , 1990, AAAI.

[8]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[9]  Yu Zheng,et al.  GeoMAN: Multi-level Attention Networks for Geo-sensory Time Series Prediction , 2018, IJCAI.

[10]  Dan Klein,et al.  Learning from measurements in exponential families , 2009, ICML '09.

[11]  Eric P. Xing,et al.  Harnessing Deep Neural Networks with Logic Rules , 2016, ACL.

[12]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[13]  Lu Feng,et al.  CityResolver: A Decision Support System for Conflict Resolution in Smart Cities , 2018, 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS).

[14]  Sameer Singh,et al.  Injecting Logical Background Knowledge into Embeddings for Relation Extraction , 2015, NAACL.

[15]  Guy Van den Broeck,et al.  Learning Logistic Circuits , 2019, AAAI.

[16]  Sanjit A. Seshia,et al.  Mining Requirements From Closed-Loop Control Models , 2015, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[17]  Meikang Qiu,et al.  Health-CPS: Healthcare Cyber-Physical System Assisted by Cloud and Big Data , 2017, IEEE Systems Journal.

[18]  Guy Van den Broeck,et al.  A Semantic Loss Function for Deep Learning with Symbolic Knowledge , 2017, ICML.

[19]  Felice Andrea Pellegrino,et al.  Model Predictive Control of glucose concentration based on Signal Temporal Logic specifications , 2019, 2019 6th International Conference on Control, Decision and Information Technologies (CoDIT).

[20]  Wen Wen,et al.  Embedding Logic Rules Into Recurrent Neural Networks , 2019, IEEE Access.

[21]  Ning Chen,et al.  Bayesian inference with posterior regularization and applications to infinite latent SVMs , 2012, J. Mach. Learn. Res..

[22]  Abdeltawab M. Hendawi,et al.  Data Sets, Modeling, and Decision Making in Smart Cities , 2019, ACM Trans. Cyber Phys. Syst..

[23]  Sanjit A. Seshia,et al.  Compositional Falsification of Cyber-Physical Systems with Machine Learning Components , 2017, NFM.

[24]  Artur S. d'Avila Garcez,et al.  Logic Tensor Networks for Semantic Image Interpretation , 2017, IJCAI.

[25]  Artur S. d'Avila Garcez,et al.  Fast relational learning using bottom clause propositionalization with artificial neural networks , 2013, Machine Learning.

[26]  Oded Maler,et al.  Robust Satisfaction of Temporal Logic over Real-Valued Signals , 2010, FORMATS.