Towards Emergent Strong Systematicity in a Simple Dynamical Connectionist Network

One of the most striking features of human language is its systematicity. This paper focuses on a fundamental, higher-order systematicity described by word classes. We investigate if such higher-order systematicity can be learned from symbol-based examples alone using a rather simple Jordan-type recurrent neural network (RNN). The network was allowed to keep a vector, which represents a suitable starting state for the production of a particular dynamic. With this representation, the network was able to reproduce particular sentences in a deterministic manner. We also show that the trained network can acquire representations of word classes to a certain extent. The analysis reveals that the higher-order grammatical constraints are realized by means of self-organized sub-symbolic dynamics. We conclude that RNNs can learn higher-order systematicity in language. However, to increase convergence robustness, additional semantic, behaviorallygrounded sources of information should be incorporated to bootstrap systematicity from semantics.