Generic Multimedia Multimodal Agents Paradigms and Their Dynamic Reconfiguration at the Architectural Level

The multimodal fusion for natural human-computer interaction involves complex intelligent architectures which are subject to the unexpected errors and mistakes of users. These architectures should react to events occurring simultaneously, and possibly redundantly, from different input media. In this paper, intelligent agent-based generic architectures for multimedia multimodal dialog protocols are proposed. Global agents are decomposed into their relevant components. Each element is modeled separately. The elementary models are then linked together to obtain the full architecture. The generic components of the application are then monitored by an agent-based expert system which can then perform dynamic changes in reconfiguration, adaptation, and evolution at the architectural level. For validation purposes, the proposed multiagent architectures and their dynamic reconfiguration are applied to practical examples, including a W3C application.

[1]  Kurt Jensen,et al.  Coloured Petri nets (2nd ed.): basic concepts, analysis methods and practical use: volume 1 , 1996 .

[2]  Alan Messer,et al.  Psi-Pervasive Services Infrastructure , 2001, TES.

[3]  Hector J. Levesque,et al.  Team Formation , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[4]  Philip R. Cohen,et al.  Something from nothing: augmenting a paper-based work practice via multimodal interaction , 2000, DARE '00.

[5]  David Frohlich,et al.  The Design Space of Interfaces , 1992 .

[6]  P. Ciancarini,et al.  Software architecture and mobility , 1998, ISAW '98.

[7]  James L. Crowley,et al.  Multi-modal tracking of faces for video communications , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Y. Bellik.  Interfaces multimodales : concepts, modeles et architectures , 1995 .

[9]  James D. Foley,et al.  The human factors of computer graphics interaction techniques , 1984, IEEE Computer Graphics and Applications.

[10]  Alexander H. Waibel,et al.  Improving connected letter recognition by lipreading , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[11]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[12]  P. Tabeling.  Multi-Level Modeling of Concurrent and Distributed Systems , 2002 .

[13]  Alan H. Bond,et al.  Readings in Distributed Artificial Intelligence , 1988 .

[14]  H. Jürgen Müller,et al.  Negotiation principles , 1996 .

[15]  石田 亨.  Real-time search for learning autonomous agents , 1997 .

[16]  David Garlan,et al.  Specifying Dynamism in Software Architectures , 1997 .

[17]  Jeff Magee,et al.  Self organising software architectures , 1996, ISAW '96.

[18]  M. Dertouzos.  THE FUTURE OF COMPUTING , 1999 .

[19]  Sharon L. Oviatt,et al.  Multimodal system processing in mobile environments , 2000, UIST '00.

[20]  Sharon L. Oviatt.  Multimodal signal processing in naturalistic noisy environments , 2000, INTERSPEECH.

[21]  Peyman Oreizy,et al.  Architecture-based runtime software evolution , 1998, Proceedings of the 20th International Conference on Software Engineering.

[22]  David Garlan,et al.  Project Aura: Toward Distraction-Free Pervasive Computing , 2002, IEEE Pervasive Comput..

[23]  Mark Weiser,et al.  Some computer science issues in ubiquitous computing , 1993, CACM.

[24]  William Buxton,et al.  Lexical and pragmatic considerations of input structures , 1983, COMG.

[25]  Mark Weiser,et al.  Some Computer Science Problems in Ubiquitous Computing , 1993 .

[26]  James M. Purtilo,et al.  The POLYLITH software bus , 1994, TOPL.

[27]  Gaetano Borriello,et al.  Next century challenges: data-centric networking for invisible computing: the Portolano project at the University of Washington , 1999, MobiCom.

[28]  Shawn D. Bird,et al.  Toward a Taxonomy of Multi-Agent Systems , 1993, Int. J. Man Mach. Stud..

[29]  Jeff Magee,et al.  Dynamic structure in software architectures , 1996, SIGSOFT '96.

[30]  Theo D'Hondt,et al.  Experiences in mobile computing: the CBorg mobile multi-agent system , 2001, Proceedings Technology of Object-Oriented Languages and Systems. TOOLS 38.

[31]  Toby Bloom,et al.  Reconfiguration and module replacement in Argus: theory and practice , 1993, Softw. Eng. J..

[32]  Michael Wooldridge,et al.  Applications of intelligent agents , 1998 .

[33]  James D. Hollan,et al.  Direct Manipulation Interfaces , 1985, Hum. Comput. Interact..

[34]  K. Ducatel,et al.  Scenarios for Ambient Intelligence in 2010 Final Report , 2001 .

[35]  Jean-Raymond Abrial,et al.  The B-book - assigning programs to meanings , 1996 .

[36]  Rocco De Nicola,et al.  KLAIM: A Kernel Language for Agents Interaction and Mobility , 1998, IEEE Trans. Software Eng..

[37]  Gerhard Weiss,et al.  Multiagent Systems , 1999 .

[38]  Kurt Jensen,et al.  Coloured Petri Nets: Basic Concepts, Analysis Methods and Practical Use. Vol. 2, Analysis Methods , 1992 .

[39]  Mario Barbacci,et al.  Durra: a structure description language for developing distributed applications , 1993, Softw. Eng. J..

[40]  Sharon L. Oviatt,et al.  Mutual disambiguation of recognition errors in a multimodel architecture , 1999, CHI '99.

[41]  Richard N. Taylor,et al.  A Component- and Message-Based Architectural Style for GUI Software , 1995, 1995 17th International Conference on Software Engineering.

[42]  Yacine Bellik,et al.  Multimodal interfaces: new solutions to the problem of computer accessibilty for the blind , 1994, CHI '94.

[43]  Jock D. Mackinlay,et al.  The design space of input devices , 1990, CHI '90.

[44]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[45]  Rangasami L. Kashyap,et al.  Temporal And Spatial Semantic Models For Multimedia Presentations , 1997 .

[46]  Jeff Magee,et al.  The Evolving Philosophers Problem: Dynamic Change Management , 1990, IEEE Trans. Software Eng..

[47]  Frédéric Azémard.  Des références dans le dialogue homme-machine multimodal. Une approche adaptée du formalisme des graphes conceptuels , 1995 .