Multimodal Representations as Basis for Cognitive Architecture

In this talk, I outline a view of “cognitive state” as fundamentally multi-modal, i.e., as an integrated and interlinked collection of “images” in various modalities: the perceptual ones, and the kinesthetic and conceptual modalities. Thinking, problem solving, reasoning, etc. are best viewed as sequences of such states, in which there is no intrinsically preferred mode. Representational elements in one mode invoke elements in other modes. The external world also at various points contributes elements to one mode or another. Perception and imagination are more continuous in this view than in the traditional views. In recent years, there has been much interest in the notion of “mental images.” However, the focus in this stream of research has been on a very special class of mental images, namely visual ones. The proposed view is an extension and generalization of this notion, not only to other perceptual modalities, but also to kinesthetic and conceptual modalities. I think the proposed view of the essential nature of the mental state opens up new ways of thinking about cognitive architecture, and also suggests new ways of building smart machines. I’ll outline why I think so.