Web-based temporal typography for musical expression and performance

This paper introduces programmable text rendering that enables temporal typography in web browsers. Typing is seen not only as a dynamic but interactive process facilitating both scripted and live musical expression in various contexts such as audio-visual performance using keyboards and live coding visualization. With the programmable text animation , we turn plain text into a highly audiovisual medium and a musical interface which is visually expressive. We describe a concrete technical realization of the concept using Web Audio API, WebGL and GLSL shaders. We further show a number of examples that illustrate instances of the concept in various scenarios ranging from simple textual visualization to live coding environments. Lastly, we present an audiovisual music piece that involves live writing augmented by the visualization technique.

[1]  Thor Magnusson.  The IXI Lang: A SuperCollider Parasite for Live Coding , 2011, ICMC.

[2]  Nick Collins,et al.  Live coding in laptop performance , 2003, Organised Sound.

[3]  Miller S. Puckette,et al.  Combining Event and Signal Processing in the MAX Graphical Programming Environment , 1991 .

[4]  Benjamin Taylor,et al.  Simplified Expressive Mobile Development with NexusUI, NexusUp, and NexusDrop , 2014, NIME.

[5]  Jason Freeman,et al.  Web-based collaboration, live musical performance and open-form scores , 2010 .

[6]  Charles Roberts,et al.  Beyond editing: extended interaction with textual code fragments , 2015, NIME.

[7]  Jonathan Berger,et al.  WAAX: Web Audio API eXtension , 2013, NIME.

[8]  JoAnn Kuchera-Morin,et al.  Gibber: Live coding audio in the Browser , 2012, ICMC.

[9]  Jason Freeman,et al.  massMobile: towards a flexible framework for large-scale participatory collaborations in live performances , 2013 .

[10]  Alex McLean,et al.  Collaboration and learning through live coding , 2014 .

[11]  Owen Kaser,et al.  Tag-Cloud Drawing: Algorithms for Cloud Visualization , 2007, ArXiv.

[12]  Chad McKinney.  Quick Live Coding Collaboration In The Web Browser , 2014, NIME.

[13]  McCartneyJames.  Rethinking the Computer Music Language , 2002 .

[14]  Matthew Wright,et al.  The Web Browser As Synthesizer And Interface , 2013, NIME.

[15]  Sang Won Lee,et al.  Real-Time Music Notation in Mixed Laptop–Acoustic Ensembles , 2013, Computer Music Journal.

[16]  Abram Hindle,et al.  SWARMED: Captive Portals, Mobile Devices, and Audience Participation in Multi-User Music Performance , 2013, NIME.

[17]  Álvaro Barbosa.  Public Sound Objects: a shared environment for networked music practice on the Web , 2005 .

[18]  Davide Della Casa,et al.  LiveCodeLab 2.0 and its language LiveCodeLang , 2014, FARM '14.

[19]  S Puckette Miller,et al.  Pure Data : another integrated computer music environment , 1996 .

[20]  Perry R. Cook,et al.  Don't forget the laptop: using native input capabilities for expressive musical control , 2007, NIME '07.

[21]  Luke Dahl,et al.  TweetDreams: Making Music with the Audience and the World using Real-time Twitter Data , 2011, NIME.

[22]  Sang Won Lee,et al.  Communication, Control, and State Sharing in Collaborative Live Coding , 2014, NIME.

[23]  Alex McLean,et al.  Visualisation of live code , 2010, EVA.

[24]  Lonce L. Wyse,et al.  The Viability of the Web Browser as a Computer Music Platform , 2013, Computer Music Journal.

[25]  Perry R. Cook,et al.  ChucK: A Concurrent, On-the-fly, Audio Programming Language , 2003, ICMC.

[26]  Boris Smus.  Web Audio API , 2013 .

[27]  久我 尚美.  Ars Electronica Festival 参加報告 , 2009 .

[28]  Andrew R. Brown,et al.  Aa-cell in Practice: an Approach to Musical Live coding , 2007, ICMC.

[29]  Phil Burk.  Jammin' on the Web - a new Client/Server Architecture for Multi-User Musical Performance , 2000, ICMC.

[30]  Scott E. Hudson,et al.  The kinetic typography engine: an extensible system for animating expressive text , 2002, UIST '02.

[31]  James McCartney,et al.  Rethinking the Computer Music Language: SuperCollider , 2002, Computer Music Journal.

[32]  John G. Hosking,et al.  Visual code annotations for cyberphysical programming , 2013, 2013 1st International Workshop on Live Programming (LIVE).