An exploratory study of multimodal interaction modeling based on neural computation

Multimodal interaction serves an important role in human-computer interaction. In this paper we propose a multimodal interaction model based on the latest cognitive research findings. The proposed model combines two proven neural computations, and helps to reveal the enhancement or depression influence of multimodal presentation upon the corresponding interaction task performance. A set of experiments is designed and conducted within the constraints of the model, which demonstrates the observed performance enhancement and depression effects. Our exploration and the experimental results help to further solve the question about how tactile feedback signal contribute the multimodal interaction efficiency which could provide guidelines for designing the tactile feedback in multimodal interaction.摘要创新点多通道交互在人机交互中具有重要的作用。 本文基于最新的认知研究成果, 提出了一种多通道交互模型。 该模型把两个已被证明的神经计算相结合, 用于揭示不同的多通道呈现对相应的交互任务绩效所产生的增强或抑制效果。 本文在该模型的适用约束下设计并实现了一组实验, 实验得出观察到的绩效增强效应和观察到的绩效抑制效应。 本文的探索思路和实验结果有助于进一步解决触觉反馈信息对多通道交互效率的贡献问题, 从而为多通道交互中触觉反馈的设计提供指导。