Integrated Multimodal Data Mining Using Self-Organizing Maps

We argue that current data mining systems have failed to keep up with the rapid increase in multimedia data available to the public and that improved content-based indexing methods should be utilized to solve this problem. This entails bridging the semantic gap between the high-level semantic concepts of humans and the low-level statistical representations us ed by computer systems. Furthermore, the problem of multimodal fusion of information coming from different data types must be solved in a multimedia scenario. PicSOM, a content-based information retrieval system described in this paper, addresses both these problems in a consistent manner by introducing multimodal object hierarchies and utilizing the strong data mining and discovery properties of Self-Organizing Maps (SOMs). We demonstrate this by providing two real-world examples. First we show how semantic associations emerge from images and subjective evaluations of personal items collected at an art installation. Secondly we review our results from the TRECVID video retrieval evaluations, and how our methodology helped in finding semantically similar objects from a large multimodal database.