SLOT: A research platform for investigating multimodal communication

In this article, we present the spatial logistics task (SLOT) platform for investigating multimodal communication between 2 human participants. Presented are the SLOT communication task and the software and hardware that has been developed to run SLOT experiments and record the participants’ multimodal behavior. SLOT offers a high level of flexibility in varying the context of the communication and is particularly useful in studies of the relationship between pen gestures and speech. We illustrate the use of the SLOT platform by discussing the results of some early experiments. The first is an experiment on negotiation with a one-way mirror between the participants, and the second is an exploratory study of automatic recognition of spontaneous pen gestures. The results of these studies demonstrate the usefulness of the SLOT platform for conducting multimodal communication research in both human-human and human-computer interactions.

[1]  Daan Broeder,et al.  The EUDICO Project, Multi Media Annotation over the Internet , 2000, LREC.

[2]  Chin-Hui Lee,et al.  DESIGN PRINCIPLES AND TOOLS FOR MULTIMODAL DIALOG SYSTEMS , 2000 .

[3]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[4]  D. Sperber,et al.  Relevance: Communication and cognition, 2nd ed. , 1995 .

[5]  S. Fiske,et al.  Social Psychology , 2019, Encyclopedia of Personality and Individual Differences.

[6]  Daan Broeder,et al.  EUDICO, Annotation and Exploitation of Multi Media Corpora over the Internet , 2000 .

[7]  M. Argyle,et al.  The Communication of Inferior and Superior Attitudes by Verbal and Non‐verbal Signals* , 1970 .

[8]  R. Krauss,et al.  Verbal, vocal, and visible factors in judgments of another's affect. , 1981 .

[9]  A. Mehrabian,et al.  Decoding of inconsistent communications. , 1967, Journal of personality and social psychology.

[10]  Dane Archer,et al.  Words and everything else: Verbal and nonverbal cues in social interpretation. , 1977 .

[11]  L. Tickle-Degnen,et al.  The Nature of Rapport and Its Nonverbal Correlates , 1990 .

[12]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[13]  H. H. Clark,et al.  Speaking while monitoring addressees for understanding , 2004 .

[14]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[15]  P. Ellsworth,et al.  Visual Behavior in Social Interaction , 1972 .

[16]  Louis Vuurpijl,et al.  Architectures for detecting and solving conflicts: two-stage classification and support vector classifiers , 2003, Document Analysis and Recognition.

[17]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[18]  D. Sperber,et al.  Relevance: Communication and Cognition , 1997 .

[19]  C. Goodwin.  Conversational Organization: Interaction Between Speakers and Hearers , 1981 .

[20]  M. Morris,et al.  Rapport in conflict resolution: Accounting for how face-to-face contact fosters mutual cooperation in mixed-motive conflicts. , 2000 .

[21]  Terri Gullickson,et al.  Social Psychology (2nd ed.). , 1995 .

[22]  Heinrich H. Bülthoff,et al.  How believable are real faces? Towards a perceptual basis for conversational animation , 2003, Proceedings 11th IEEE International Workshop on Program Comprehension.

[23]  C. Hendrick.  The Nature of Rapport , 1990 .

[24]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[25]  A. Kendon.  Some Relationships Between Body Motion and Speech , 1972 .

[26]  D. McNeill.  Hand and Mind , 1995 .

[27]  J. D. Ruiter.  The production of gesture and speech , 2000 .

[28]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[29]  Anne H. Anderson,et al.  The Hcrc Map Task Corpus , 1991 .