Learning Semantics of Movement

The basic scientific question behind this presentation is how to model computationally the interrelated processes of understanding natural language and perceiving and producing movement in real world contexts. Rather than relying on manually defined representations, we approach this problem by analyzing the statistical regularities in contexts of language use. A central aspect is that the learning of internal representations is based on large amounts of fully or mostly unlabeled training data [1, 2]. A promising methodological approach towards this is that data from different sources can be brought together through multiview/transfer/multi-task learning (see e.g. [2, 3, 4, 5]). Another promising approach is learning relations. Relations are powerful abstractions because they allow reasoning not just about objects, but about their combinations. Recent work suggests that at least limited classes of relations can be learned from data [6]. We have earlier conducted research in which text contexts have been used to learn semantic similarities (see e.g. [7, 8]. However, to reach a more human-level understanding, we have to take into account that language is fully understood only through its use in its multimodal and embodied contexts including linguistic, visual, auditory, tactile and kinestetic dimensions. In general, patterns and signals are natural representations for multimodal contexts. These differ considerably from the discrete representations of symbols and expressions in symbolic languages. For instance, images are typically represented as numerical matrices. We have earlier conducted research related to the combination of image and language data (see e.g. [9, 10]).