Use of Sound with Digital Text: Moving beyond Sound as an Add-On or Decoration.

The purpose of this interpretive case study was to explore—through a close analysis of one class project—students’ use of audio signs and the teacher's scaffolding of the use of audio signs. Two research questions guided this study: (a) In what ways did the fifth-grade students use audio signs, specifically transitions sounds, when constructing multimodal texts with different sign systems (e.g., visual, linguistic or audio signs)?; (b) In what ways did the classroom teacher shape the specific social cultural environment for audio sign use? The findings of this study argue for professional development opportunities for teachers where they not only learn how to use various software programs but also learn the content knowledge necessary for communicating with multiple signs such as audio.

[1]  Colin Lankshear,et al.  Sampling "the new" in New Literacies , 2007 .

[2]  Gunther Kress,et al.  Visual and verbal modes of representation in electronically mediated communication: the potentials of new forms of text , 1998 .

[3]  Emmanuel Bigand,et al.  Contributions of music to research on human auditory cognition. , 1993 .

[4]  Lynn E. Shanahan,et al.  Using Digital Media to Interpret Poetry: Spiderman Meets Walt Whitman. , 2008 .

[5]  Diana Deutsch,et al.  Auditory pattern recognition. , 1986 .

[6]  William W. Gaver.  What in the World Do We Hear? An Ecological Approach to Auditory Event Perception , 1993 .

[7]  Asim Evren Yantaç,et al.  The effects of the sound-image relationship within sound education for interactive media design , 2006, Digit. Creativity.

[8]  G. Kress,et al.  Multimodal Discourse: The Modes and Media of Contemporary Communication , 2002 .

[9]  Jabari Mahiri.  Digital DJ-ing: Rhythms of Learning in an Urban School. , 2006 .

[10]  B. Cope,et al.  Multiliteracies: Literacy Learning and the Design of Social Futures , 2001 .

[11]  G. Hull,et al.  Locating the Semiotic Power of Multimodality , 2005 .

[12]  Lynn E. Shanahan.  Reading and writing multimodal texts through information and communication technologies , 2006 .

[13]  R. Altman,et al.  Sound theory, sound practice , 1992 .

[14]  L. Kaufman,et al.  Handbook of perception and human performance , 1986 .

[15]  A. Rip.  Science for the 21st Century , 2002 .

[16]  Malcolm L. Fleming,et al.  Instructional Message Design: Principles from the Behavioral and Cognitive Sciences , 1993 .

[17]  D L Kohfeld,et al.  Simple reaction time as a function of stimulus intensity in decibels of light and sound. , 1971, Journal of experimental psychology.

[18]  Michael Joyce,et al.  Page to Screen: Taking Literacy into the Electronic Age , 1998 .

[19]  Suzanne M. Miller,et al.  Teacher Learning for New Times: Repurposing New Multimodal Literacies and Digital-Video Composing for Schools , 2007 .

[20]  M. Kalantzis,et al.  Designs for social futures , 2005 .

[21]  Jamie Myers,et al.  Hypermedia Authoring as Critical Literacy. , 2001 .

[22]  S. Michaels,et al.  A pedagogy of Multiliteracies Designing Social Futures , 1996 .

[23]  C. Jewitt.  Multimodality, “Reading”, and “Writing” for the 21st Century , 2005 .

[24]  Kevin Leander.  Composing with Old and New Media: Toward a Parallel Pedagogy , 2009 .

[25]  G. Kress,et al.  Writing in Multimodal Texts , 2008 .

[26]  James Paul Gee,et al.  What video games have to teach us about learning and literacy , 2007, CIE.

[27]  K. Zammit.  The New Learning Environments framework: scaffolding the development of multiliterate students , 2010 .

[28]  J. Blommaert,et al.  CRITICAL DISCOURSE ANALYSIS , 2000 .

[29]  Thomas D. Snyder,et al.  Digest of Education Statistics, 2005. NCES 2006-030. , 2006 .

[30]  Sharan B. Merriam,et al.  Qualitative research and case study applications in education , 1998 .

[31]  M. Halliday.  Language as social semiotic: The social interpretation of language and meaning , 1976 .

[32]  T. V. Leeuwen.  Speech, Music, Sound , 1999 .

[33]  William W. Gaver.  Synthesizing auditory icons , 1993, INTERCHI.

[34]  Colin Lankshear,et al.  A new literacies sampler , 2007 .

[35]  L. Murray.  Authenticity and realism in documentary sound , 2010 .

[36]  Michel Chion,et al.  Audio-Vision: Sound on Screen , 1994 .

[37]  L. Miller,et al.  The impact of new information technology on young children's symbol-weaving efforts , 1992 .

[38]  Ward Mitchell Cates,et al.  Sound’s use in instructional software to enhance learning: a theory-to-practice content analysis , 2008 .

[39]  Mary Kalantzis,et al.  Learning by Design , 2010 .

[40]  Thomas D. Snyder,et al.  Digest of Education Statistics , 1994 .

[41]  Jason Ranker.  Learning Nonfiction in an ESL Class: The Interaction of Situated Practice and Teacher Scaffolding in a Genre Study. , 2009 .

[42]  J. Kelly,et al.  Sensing the world , 1983 .

[43]  Marina R Y Gall,et al.  Music composition lessons: the multimodal affordances of technology , 2005 .

[44]  Jason Ranker,et al.  Making Meaning on the Screen: Digital Video Production About the Dominican Republic , 2008 .