The thematic modelling of subtext

Narratives form a key component of multimedia knowledge representation on the Web. However, many existing multimedia narrative systems either ignore the narrative qualities of any media, or focus on the literal depicted content ignoring any subtext. Ignoring narrative subtext can lead to erroneous search results, or automatically remixed content that lacks cohesion. We suggest that subtext can be computationally modeled in terms of Tomashevsky’s hierarchy of themes and motifs. These elements can then be used in a semiotic term expansion algorithm, incorporating knowledge of subtext into search and subsequent narrative generation. We present two experimental applications of this technique. In the first, we use our thematic model in the automatic construction of photo montages from Flickr, comparing it to more traditional term expansion based on co-occurrence, and showing that this improves the perceived relevance of images within the montage. In the second, we use the thematic model in order to automatically identify Flickr images to illustrate short stories, where it dampened the perception of unwanted themes (an effect we describe as reducing thematic noise). Our work is among the first in this space, and shows that thematic subtext can be tackled computationally.