Integrating Domain-Knowledge into Deep Learning

In this talk, I will first discuss deep learning models that can find semantically meaningful representations of words, learn to read documents and answer questions about their content. I will show how we can encode external linguistic knowledge as an explicit memory in recurrent neural networks, and use it to model co-reference relations in text. I will further introduce methods that can augment neural representation of text with structured data from Knowledge Bases for question answering, and show how we can use structured prior knowledge from Knowledge Graphs for image classification. Finally, I will introduce the notion of structured memory as being a crucial part of an intelligent agent's ability to plan and reason in partially observable environments. I will present a modular hierarchical reinforcement learning agent that can learn to store arbitrary information about the environment over long time lags, perform efficient exploration and long-term planning, while generalizing across domains and tasks.