Multimodal Interfaces of Human–Computer Interaction

An analytical review of state-of-the-art and future intelligent interfaces of human–computer interaction is presented; stages of their evolution are considered from command text to graphic and then to intelligent uni- and multimodal interfaces, based on the transfer of acoustic, visual, textual, and neural information. The principles of organization and the main characteristics and types of multimodal user interfaces, which employ concurrently several tools for automatic processing (recognition and synthesis) of userinputted heterogeneous information, are detailed. The combination of computers with speech and multimodal interfaces, designed for user-friendly information input/output, creates universal information–communicative technologies, man coming to the fore in the interaction between man and computer. Russian and foreign developments in this field are analyzed briefly.

[1]  Tetsuya Ogata,et al.  Audio-visual speech recognition using deep learning , 2014, Applied Intelligence.

[2]  A. A. Karpov,et al.  Information enquiry kiosk with multimodal user interface , 2009, Pattern Recognition and Image Analysis.

[3]  Darryl Stewart,et al.  Robust Audio-Visual Speech Recognition Under Noisy Audio-Video Conditions , 2014, IEEE Transactions on Cybernetics.

[4]  Denham L. Phipps.  The human–computer interaction handbook: fundamentals, evolving technologies and emerging applications (3rd ed) , 2013 .

[5]  Alexey A. Karpov.  An automatic multimodal speech recognition system with audio and video information , 2014, Autom. Remote. Control..

[6]  Andrey Ronzhin,et al.  АНАЛИЗ МЕТОДОВ МНОГОМОДАЛЬНОГО ОБЪЕДИНЕНИЯ ИНФОРМАЦИИ ДЛЯ АУДИОВИЗУАЛЬНОГО РАСПОЗНАВАНИЯ РЕЧИ , 2016 .

[7]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[8]  Matthew Turk,et al.  Multimodal interaction: A review , 2014, Pattern Recognit. Lett..

[9]  Raymond S. Nickerson,et al.  Human interaction with computers and robots , 1995 .

[10]  Ирина Сергеевна Кипяткова,et al.  Разновидности глубоких искусственных нейронных сетей для систем распознавания речи , 2016 .

[11]  Eric Horvitz,et al.  Facilitating multiparty dialog with gaze, gesture, and speech , 2010, ICMI-MLMI '10.

[12]  Sharon Oviatt,et al.  Multimodal Interfaces , 2008, Encyclopedia of Multimedia.

[13]  A KarpovAlexey,et al.  BILINGUAL MULTIMODAL SYSTEM FOR TEXT-TO-AUDIOVISUAL SPEECH AND SIGN LANGUAGE SYNTHESIS , 2014 .

[14]  Dimitrios Tzovaras.  Multimodal user interfaces : from signals to interaction , 2008 .

[15]  Mahesh S. Raisinghani,et al.  Ambient Intelligence: Changing Forms of Human-Computer Interaction and their Social Implications , 2006, J. Digit. Inf..

[16]  Eric Vatikiotis-Bateson,et al.  Audiovisual Speech Processing: Contributors , 2012 .

[17]  Ronzhin,et al.  Multimodal Interfaces: Main Principles and Cognitive Aspects@@@Многомодальные интерфейсы: основные принципы и когнитивные аспекты , 2014 .

[18]  Michael Johnston,et al.  Articles: Robust Understanding in Multimodal Interfaces , 2009, CL.

[19]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[20]  Philippe A. Palanque,et al.  Fusion engines for multimodal input: a survey , 2009, ICMI-MLMI '09.

[21]  Vera Kaiser,et al.  Thinking Penguin: Multimodal Brain–Computer Interface Control of a VR Game , 2013, IEEE Transactions on Computational Intelligence and AI in Games.

[22]  Andrey Ronzhin,et al.  From smart devices to smart space , 2010 .

[23]  Stefano Federici,et al.  Assistive technology assessment handbook , 2017 .

[24]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[25]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[26]  Anton Nijholt,et al.  Experiencing BCI Control in a Popular Computer Game , 2013, IEEE Transactions on Computational Intelligence and AI in Games.

[27]  Aggelos K. Katsaggelos,et al.  Audiovisual Fusion: Challenges and New Approaches , 2015, Proceedings of the IEEE.