A comparative analysis of multimodal communication during design sketching in co-located and distributed environments

This study extends our understanding of multimodal communication during design sketching. Building on the literature, the theoretical dimension frames gesturing as a communication channel and a thinking medium, and postulates an interplay between gesturing and other channels. The empirical dimension explores the theoretical propositions in the context of co-located and distributed sketching. Quantitative analyses suggest that when gesturing is restricted, graphical communication is leveraged to compensate, and that verbal communication is incessant in both collaboration environments. They also highlight a non-compensatory design phase dependent interaction between gestural and graphical communication. Moreover, they reveal differences in the communication structure used in the two environments. Qualitative analyses identify a behavior termed “cross-gesturing,” which informs how gesturing facilitates shared understanding during collaborative sketching.

[1]  Barbara Tversky,et al.  Explanations in Gesture, Diagram, and Word , 2009, Spatial Language and Dialogue.

[2]  S. Mohammed,et al.  Mental models in design teams: a valid approach to performance in design collaboration? , 2007 .

[3]  J. Cassell.  Computer Vision for Human–Machine Interaction: A Framework for Gesture Generation and Interpretation , 1998 .

[4]  Masaki Suwa,et al.  What do architects and students perceive in their design sketches? A protocol analysis , 1997 .

[5]  I. Janis.  Groupthink: Psychological Studies of Policy Decisions and Fiascoes , 1982 .

[6]  Willemien Visser,et al.  The Function of Gesture in an Architectural Design Meeting , 2009, ArXiv.

[7]  R. Hall,et al.  Members of great teams think alike: A model of team effectiveness and schema similarity among team members. , 1994 .

[8]  John S. Gero,et al.  Drawings and the design process , 1998 .

[9]  Sara A. Bly,et al.  A use of drawing surfaces in different collaborative settings , 1988, CSCW '88.

[10]  Willemien Visser,et al.  The role of gesture in designing , 2011, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[11]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[12]  Robert Ramberg,et al.  Where all the interaction is: sketching in interaction design as an embodied practice , 2008, DIS '08.

[13]  John C. Tang,et al.  A framework for understanding the workspace activity of design teams , 1988, CSCW '88.

[14]  Gerry Stahl,et al.  Group Cognition: Computer Support for Building Collaborative Knowledge (Acting with Technology) , 2006 .

[15]  Newton D'Souza,et al.  The metaphor of an ensemble: architectural design and skill convergence , 2012 .

[16]  Eugene S. Ferguson,et al.  Engineering and the Mind's Eye , 1994 .

[17]  John S. Gero,et al.  Comparing collaborative co-located and distributed design processes in digital and traditional sketching environments: A protocol study using the function-behaviour-structure coding scheme , 2011 .

[18]  S. Goldin-Meadow,et al.  Hearing Gesture: How Our Hands Help Us Think , 2003 .

[19]  Jacob Buur,et al.  Getting the point: The role of gesture in managing intersubjectivity in a design activity , 2011, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[20]  L.J. Leifer,et al.  Engineering design thinking, teaching, and learning , 2005, IEEE Engineering Management Review.

[21]  Holly Arrow,et al.  Small Groups as Complex Systems: Formation, Coordination, Development, and Adaptation , 2000 .

[22]  S. Goldin-Meadow,et al.  The role of gesture in communication and thinking , 1999, Trends in Cognitive Sciences.

[23]  Sheryl Staub-French,et al.  Physical and Digital Artifact-Mediated Coordination in Building Design , 2008, Computer Supported Cooperative Work (CSCW).

[24]  Hiroshi Ishii,et al.  ClearBoard: a seamless medium for shared drawing and conversation with eye contact , 1992, CHI.

[25]  Willemien Visser,et al.  The Cognitive Artifacts of Designing , 2006 .

[26]  John C. Tang,et al.  VideoWhiteboard: video shadows to support remote collaboration , 1991, CHI.

[27]  John C. Tang,et al.  VideoDraw: a video interface for collaborative drawing , 1990, CHI '90.

[28]  Christoph Meinel,et al.  Analyzing distributed whiteboard interactions , 2011, 7th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom).

[29]  Remko van der Lugt,et al.  How sketching can affect the idea generation process in design group meetings , 2005 .

[30]  Eiko Yasui,et al.  Collaborative idea construction: Repetition of gestures and talk in joint brainstorming , 2013 .

[31]  Judith S. Olson,et al.  Analysis of gestures in face-to-face design teams provides guidance for how to use groupware in design , 1995, Symposium on Designing Interactive Systems.

[32]  Paul Rodgers,et al.  Visible ideas:Information patterns of conceptual sketch activity , 1998 .

[33]  S. Mohammed,et al.  Team Mental Model: Construct or Metaphor? , 1994 .

[34]  Roger Bakeman,et al.  Observer agreement for timed-event sequential data: A comparison of time-based and event-based algorithms , 2009, Behavior research methods.

[35]  Susan R. Fussell,et al.  Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks , 2004, Hum. Comput. Interact..

[36]  V. Goel.  Sketches of thought , 1995 .

[37]  Keith M. Murphy,et al.  Collaborative imagining: The interactive use of gestures, talk, and graphic representation in architectural practice , 2005 .

[38]  A. Dong,et al.  Investigating design cognition in the construction and enactment of team mental models , 2013 .

[39]  Nicolas Roussel,et al.  Exploring New Uses of Video with VideoSpace , 2001, EHCI.

[40]  Hiroshi Ishii,et al.  Toward an open shared workspace: computer and video fusion approach of TeamWorkStation , 1991, CACM.

[41]  Petra Badke-Schaub,et al.  Conversations around Design Sketches: Use of Communication Channels for Sharing Mental Models during Concept Generation. , 2012 .

[42]  P. Ekman,et al.  The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding , 1969 .

[43]  D. McNeill.  Hand and Mind: What Gestures Reveal about Thought , 1992 .

[44]  Gerhard Fischer,et al.  Transcending the individual human mind—creating shared understanding through collaborative design , 2000, TCHI.

[45]  John S. Gero,et al.  The Situated Function — Behaviour — Structure Framework , 2004 .