A multi-modal network architecture for knowledge discovery

The collection and assessment of national security related information often involves an arduous process of detecting relevant associations between people, events, and locations—typically within very large data sets. The ability to more effectively perceive these connections could greatly aid in the process of knowledge discovery. This same process—pre-consciously collecting and associating multimodal information—naturally occurs in mammalian brains. With this in mind, this effort sought to draw upon the neuroscience community’s understanding of the relevant areas of the brain that associate multi-modal information for long-term storage for the purpose of creating a more effective, and more automated, association mechanism for the analyst community. Using the biology and functionality of the hippocampus as an analogy for inspiration, we have developed an artificial neural network architecture to associate k-tuples (paired associates) of multimodal input records. The architecture is composed of coupled unimodal self-organizing neural modules that learn generalizations of unimodal components of the input record. Cross modal associations, stored as a higher-order tensor, are learned incrementally as these generalizations are formed. Graph algorithms are then applied to the tensor to extract multi-modal association networks formed during learning. Doing so yields a potential novel approach to data mining for intelligence-related knowledge discovery. This paper describes the neurobiology, architecture, and operational characteristics, as well as provides a simple intelligence-based example to illustrate the model’s functionality.