A multimodal approach to examining 'embodiment' in tangible learning environments

Tangible and multitouch technologies offer new opportunities for physically interacting with objects and digital representations, foregrounding the role of the body in interaction and learning. Developing effective methodologies for examining real time cognition and action, and the relationship between embodiment, interaction and learning is challenging. This paper draws on multimodality, with its emphasis on examining the use of multiple semiotic resources for meaning making, to examine the differential use of semiotic resources by pairs of students interacting with a tangible learning environment. Specifically the analysis details the role of body position, gaze, manipulation and speech in shaping interaction. The analysis illustrates the interaction between these modes and 'multimodal action flow', particularly in terms of pace, rhythm and interaction structure, and the implications of this for interaction and the meaning making process.

[1]  Eva Insulander.  Book Review: Multimodality: A Social Semiotic Approach to Communication , 2011 .

[2]  Sara Price,et al.  What have you done! the role of 'interference' in tangible environments for supporting collaborative learning , 2009, CSCL.

[3]  Paul P. Maglio,et al.  On Distinguishing Epistemic from Pragmatic Action , 1994, Cogn. Sci..

[4]  Paul Marshall,et al.  Beyond one-size-fits-all: how interactive tabletops support collaborative learning , 2011, IDC.

[5]  Yvonne Rogers,et al.  Children designing together on a multi-touch tabletop: an analysis of spatial orientation and user interactions , 2009, IDC.

[6]  Scott R. Klemmer,et al.  How bodies matter: five themes for interaction design , 2006, DIS '06.

[7]  Jennifer G. Sheridan,et al.  DIY design process for interactive surfaces , 2009, BCS HCI.

[8]  Sara Price,et al.  Towards a framework for investigating tangible environments for learning , 2008, Int. J. Arts Technol..

[9]  R. Fawcett.  Language as social semiotic , 1984 .

[10]  Linda Jarvin,et al.  When Theories Don't Add Up: Disentangling he Manipulatives Debate , 2007 .

[11]  Reed Stevens,et al.  Scaling practices of spatial analysis and modeling , 2010 .

[12]  Yvonne Rogers,et al.  Actions speak loudly with words: unpacking collaboration around the table , 2009, ITS '09.

[13]  Andrew Burn,et al.  Making New Media: Creative Production and Digital Literacies , 2009 .

[14]  Carey Jewitt,et al.  Technology, literacy and learning: a multimodal approach , 2010 .

[15]  Yvonne Rogers,et al.  Around the table: are multiple-touch surfaces better than single-touch for children's collaborative interactions? , 2009, CSCL.

[16]  Sara Price,et al.  Interfering and resolving: How tabletop interaction facilitates co-construction of argumentative knowledge , 2011, Int. J. Comput. Support. Collab. Learn..

[17]  Charles Goodwin,et al.  Embodied Interaction: Language And Body In The Material World , 2014 .

[18]  Hiroshi Ishii,et al.  Illuminating light: an optical design tool with a luminous-tangible interface , 1998, CHI.

[19]  C. Jewitt.  The Routledge handbook of multimodal analysis , 2014 .

[20]  Paul Marshall,et al.  Editorial: the evolving field of tangible interaction for children: the challenge of empirical validation , 2012, Personal and Ubiquitous Computing.

[21]  Greg Corness,et al.  Playing with the sound maker: do embodied metaphors help children learn? , 2008, IDC.