Temporal and Object Quantification Networks

We present Temporal and Object Quantification Networks (TOQ-Nets), a new class of neuro-symbolic networks with a structural bias that enables them to learn to recognize complex relational-temporal events. This is done by including reasoning layers that implement finite-domain quantification over objects and time. The structure allows them to generalize directly to input instances with varying numbers of objects in temporal sequences of varying lengths. We evaluate TOQ-Nets on input domains that require recognizing event-types in terms of complex temporal relational patterns. We demonstrate that TOQ-Nets can generalize from small amounts of data to scenarios containing more objects than were present during training and to temporal warpings of input sequences.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Chong Wang,et al.  Neural Logic Machines , 2019, ICLR.

[3]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[4]  Dahua Lin,et al.  Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition , 2018, AAAI.

[5]  Chris L. Baker,et al.  Rational quantitative attribution of beliefs, desires and percepts in human mentalizing , 2017, Nature Human Behaviour.

[6]  Abhinav Gupta,et al.  Videos as Space-Time Region Graphs , 2018, ECCV.

[7]  Aaron F. Bobick,et al.  A Framework for Recognizing Multi-Agent Action from Visual Evidence , 1999, AAAI/IAAI.

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Michal Koperski,et al.  Toyota Smarthome: Real-World Activities of Daily Living , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Luc Van Gool,et al.  stagNet: An Attentive Semantic RNN for Group Activity Recognition , 2018, ECCV.

[11]  Artur S. d'Avila Garcez,et al.  A Connectionist Cognitive Model for Temporal Synchronisation and Learning , 2007, AAAI.

[12]  Alan Fern,et al.  Probabilistic event logic for interval-based event recognition , 2011, CVPR 2011.

[13]  Robin R. Vallacher,et al.  What do people think they're doing? Action identification and human behavior. , 1987 .

[14]  Greg Mori,et al.  Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Greg Mori,et al.  A Hierarchical Deep Temporal Model for Group Activity Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  S. Crawford,et al.  Volume 1 , 2012, Journal of Diabetes Investigation.

[17]  Jeffrey M. Zacks,et al.  Event perception: a mind-brain perspective. , 2007, Psychological bulletin.

[18]  Dmitry Berenson,et al.  Explaining Multi-stage Tasks by Learning Temporal Logic Formulas from Suboptimal Demonstrations , 2020, Robotics: Science and Systems.

[19]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[20]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Larry S. Davis,et al.  Event Modeling and Recognition Using Markov Logic Networks , 2008, ECCV.

[22]  Andrew J. Davison,et al.  RLBench: The Robot Learning Benchmark & Learning Environment , 2019, IEEE Robotics and Automation Letters.

[23]  Alberto Camacho,et al.  Finite LTL Synthesis as Planning , 2018, ICAPS.

[24]  B. Hommel The Perception of Action and Movement , 1996 .

[25]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[26]  Fei-Fei Li,et al.  Learning latent temporal structure for complex event detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Jeffrey M. Zacks,et al.  Perceiving, remembering, and communicating structure in events. , 2001, Journal of experimental psychology. General.

[28]  Loyal Jones,et al.  Remembering , 1975, IEEE Power Engineering Review.

[29]  Daniel Neider,et al.  Learning Linear Temporal Properties , 2018, 2018 Formal Methods in Computer Aided Design (FMCAD).