Developing a language of interactivity through the theory of play

In a world increasingly influenced by interactive interfaces, devices and services both in the commercial and non-commercial spheres, understanding interactivity and its underpinnings is essential. We have moved into a state of flux in which both culture and technology are in constant interplay and the only constant is change itself. The only future-proof approach to designing for and dealing with an environment of constant change in these systems and forms is to look for a mechanism and theoretical framework that underpins them all. Drawing upon a range of disciplines – from design, art, cognitive science, linguistics and more – this thesis argues that play is such a fundamental building block of culture, society, technology and cognition that it is the ideal lens through which to examine the interactive experience. It is versatile enough to cross boundaries and fundamental enough to be understood intuitively. Through an understanding of the intersection between movement, embodied cognition, metaphor and play, a set of principles of interactivity are developed that are flexible enough to analyse and be applied to a broad spectrum of interactive experiences, from interactive artworks to services to individual user interface elements. Finally, it is proposed that these principles provide a way to examine the phenomenal growth of social networks and the fundamental cultural shifts we are experiencing today as a result of the friction generated between emerging networked technologies and the industrial age structures they are dismantling. ii ©2010 ANDREW POLAINE – PhD Thesis, Faculty of Arts & Social Sciences, University of Technology, Sydney. CONTENTS ACKNOWLEDGEMENTS i ........................................................................... ABSTRACT ii .......................................................................................... PART I PLAY, FLOW, METAPHOR & COGNITION 1 ....................................... Introduction 1 .......................................................................................... What is interactivity? 6 ........................................................................... Interactivity and the HCI field 10 ............................................................... Towards a language of interactivity 11 ......................................................... CHAPTER 1 The Role of Play in Interactivity 13 ................................................ Play as a design process, interactivity as content 14 ....................................... Playful interaction and direct manipulation 17 ................................................ Cameras and the body as affordance 20 ................................................... CHAPTER 2 – Flow, Interactivity & Suspension of Disbelief 23 .............................. Interactive Flow versus Narrative Flow 23 ................................................... Willing Suspension of Disbelief and Interactivity 25 ....................................... CHAPTER 3 Interactivity, Metaphors and the Mind 28 ....................................... The role of metaphors 28 ........................................................................ Interfaces are both metaphorical and physical 30 .......................................... A metaphor too far 36 ........................................................................... Direct control and manipulation in interfaces 40 ............................................. CHAPTER 4 Thinking Outside of the Brain 43 ................................................... Scripts, Schemata, Mental Models and Automotive Action 43 ........................... Scripts 44 ............................................................................................. Schemata 47 ....................................................................................... The embodiment of AI 51 ........................................................................ Pay no attention to that man behind the curtain 53 ....................................... Are we automata? 54 .............................................................................. Not today, I’m not in the mood 56 ............................................................ The Mojave Experiment 58 ..................................................................... PART II DEVELOPING THE PRINCIPLES OF INTERACTIVITY 63 ........................... CHAPTER 5 – The State of Play 63 .................................................................. Homo Ludens 63 ................................................................................. Les Jeux et les Hommes 65 ..................................................................... Love is... 75 .......................................................................................... From Physics Lab to Living Room to the Bus 77 .......................................... Game Studies 82 ................................................................................. The Magic Circle 89 .............................................................................. CHAPTER 6 – Principle 1: The Invitation to Play 91 ............................................. Form, function, medium and message 92 ................................................... Case Study: Time Sketches 103 .................................................................. Case Study: Body Movies – public play and interaction 110 .............................. iii ©2010 ANDREW POLAINE – PhD Thesis, Faculty of Arts & Social Sciences, University of Technology, Sydney. CHAPTER 7 – Principle 2: The Playing Field & the Rules 114 .................................... The Rules of Play 115 .............................................................................. The Lusory Attitude 116 ........................................................................... Using the lens of gameplay 118 .................................................................. We are all hackers 121 .............................................................................. CHAPTER 8 – Principle 3: Challenge, Boredom and Anxiety 123 .............................. The border between boredom and anxiety 127 ............................................. CHAPTER 9 – Principle 4: Triviality, Open-endedness, Promises 133 ........................ Playing in the Gallery 133 ........................................................................... Delivering the promise 136 ........................................................................ Case Study: Eavesdrop – an opportunity missed 140 ....................................... CHAPTER 10 – Social Interaction and Playing with Friends 145 .............................. APIs as invitations 147 .............................................................................. PART III Conclusion 151 .............................................................................. CHAPTER 11 Understanding Interactivity Through Play 151 ................................. POSTSCRIPT 155 ....................................................................................... Social Play & Cultural Disruption 155 .................................................................. REFERENCES 162 ....................................................................................... Appendix A – Time Sketches Research Interview Instructions 176 ........................... Interview Questions 178 ........................................................................... Appendix B – Time Sketches Research Interviews 180 .......................................... Subject 1 Interview 180 ........................................................................... Subject 2 Retrospective Report 182 ......................................................... Subject 2 Interview 183 ........................................................................... Subject 3 Retrospective Report 185 ......................................................... Subject 3 Interview 186 ........................................................................... Subject 4 Retrospective Report 188 ......................................................... Subject 4 Interview 190 ........................................................................... Subject 5 Retrospective Report 193 ......................................................... Subject 5 Interview 194 ........................................................................... Subject 6 Retrospective Report 196 ......................................................... Subject 6 Interview 198 ........................................................................... Subjects 7 & 8 Retrospective Report 201 ................................................... Subjects 7 & 8 Interview 205 .................................................................. iv ©2010 ANDREW POLAINE – PhD Thesis, Faculty of Arts & Social Sciences, University of Technology, Sydney.

[1]  E. Reed The Ecological Approach to Visual Perception , 1989 .

[2]  Lucy Bullivant,et al.  Responsive Environments: Architecture, Art and Design , 2006 .

[3]  Thorsten Klooster,et al.  Smart Surfaces - and their Application in Architecture and Design , 2009 .

[4]  Xavier Sarrate Donaire Don't make me think: a common sense approach to web usability.2nd ed. , 2009 .

[5]  Alessandro Soro,et al.  Multitouch Sensing for Collaborative Interactive Walls , 2008, HCIS.

[6]  D. Hoffman,et al.  Measuring the Flow Experience Among Web Users , 2000 .

[7]  Yoav Sterman,et al.  Adding Playful Interaction to Public Spaces , 2005, INTETAIN.

[8]  D. Levine,et al.  A neuropsychological theory of positive affect and its influence on cognition. , 1999, Psychological review.

[9]  Lisa A. Ennis Dark Fiber: Tracking Critical Internet Culture , 2003, J. Assoc. Inf. Sci. Technol..

[10]  K. A. Ericsson,et al.  Protocol analysis: Verbal reports as data, Rev. ed. , 1993 .

[11]  Ben Shneiderman,et al.  Direct Manipulation: A Step Beyond Programming Languages , 1983, Computer.

[12]  Jeffrey Shaw,et al.  Jeffrey Shaw-a user's manual : from expanded cinema to virtual reality = Jeffrey Shaw-eine gebrauchsanweisung : vom expanded cinema zur virtuellen realität , 1997 .

[13]  M. Csíkszentmihályi,et al.  Optimal experience: Psychological studies of flow in consciousness. , 1988 .

[14]  Marie-Laure Ryan,et al.  Beyond Myth and Metaphor - The Case of Narrative in Digital Media , 2001, Game Stud..

[15]  Stanley J. Watson,et al.  Socially-induced brain ‘fertilization’: play promotes brain derived neurotrophic factor transcription in the amygdala and dorsolateral frontal cortex in juvenile rats , 2003, Neuroscience Letters.

[16]  Wieslaw Zielonka,et al.  An Invitation to Play , 2005, MFCS.

[17]  Cliff Lampe,et al.  The Benefits of Facebook "Friends: " Social Capital and College Students' Use of Online Social Network Sites , 2007, J. Comput. Mediat. Commun..

[18]  Alison Harvey,et al.  You mean It’s only a game? Rule structures, the magic circle, and player participation in pervasive mobile gaming , 2007 .

[19]  S. Vereza Philosophy in the flesh: the embodied mind and its challenge to Western thought , 2001 .

[20]  Charles S. Carver,et al.  Some ways in which goals differ and some implications of those differences. , 1996 .

[21]  E. Edmonds,et al.  The Creative Process Where the Artist is Amplified or Superseded by the Computer , 1973 .

[22]  斎藤 香,et al.  Friendship , 1998, Terror Capitalism.

[23]  R. A. Brooks,et al.  Intelligence without Representation , 1991, Artif. Intell..

[24]  M. F. Wiser,et al.  When heat and temperature were one , 2014 .

[25]  R. Hepburn,et al.  BEING AND TIME , 2010 .

[26]  Henry Jenkins Game Design as Narrative Architecture , 2003 .

[27]  G. Lakoff,et al.  Why cognitive linguistics requires embodied realism , 2002 .

[28]  Leonard Talmy,et al.  Force Dynamics in Language and Cognition , 1987, Cogn. Sci..

[29]  Espen Aarseth Genre trouble : narrativism and the art of simulation , 2004 .

[30]  Dan Saffer,et al.  Designing for Interaction: Creating Smart Applications and Clever Devices , 2006 .

[31]  Maurice Merleau-Ponty Phenomenology of Perception , 1964 .

[32]  Rodney A. Brooks,et al.  Achieving Artificial Intelligence through Building Robots , 1986 .

[33]  Wendell Bell,et al.  The Third Wave. , 1982 .

[34]  Terry Regier,et al.  The Human Semantic Potential: Spatial Language and Constrained Connectionism , 1996 .

[35]  Eric Matthews The Philosophy of Merleau-Ponty , 2002 .

[36]  Jefferson Y. Han Multi-touch interaction wall , 2006, SIGGRAPH '06.

[37]  Robert McKee,et al.  Story: Substance, Structure, Style, and the Principles of Screenwriting , 1997 .

[38]  Nick Yee,et al.  The Labor of Fun , 2006, Games Cult..

[39]  Daniel Rozin,et al.  Wooden mirror , 2001 .

[40]  Nathan Shedroff Experience Design 1 , 2001 .

[41]  Jan Krikke Samurai Romanesque, J2ME, and the Battle for Mobile Cyberspace , 2003, IEEE Computer Graphics and Applications.

[42]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[43]  Cindy Poremba,et al.  Critical Potential on the Brink of the Magic Circle , 2007, DiGRA Conference.

[44]  Martin H. Levinson Creativity: Flow and the Psychology of Discovery and Invention , 1997 .

[45]  Jakob Nielsen,et al.  E-Commerce User Experience , 2001 .

[46]  Brenda Laurel,et al.  Computers as theatre , 1991 .

[47]  Ernest A. Edmonds,et al.  Creating Affective Visualisations for a Physiologically Interactive Artwork , 2006, Tenth International Conference on Information Visualisation (IV'06).

[48]  J. Huizinga Homo Ludens: A Study of the Play-Element in Culture , 1938 .

[49]  M. Washburn,et al.  The Play of Man. , 1902 .

[50]  Steven Johnson,et al.  Interface Culture: How New Technology Transforms the Way We Create and Communicate , 1997 .

[51]  G. Boulton‐Lewis Teaching for quality learning at university , 2008 .

[52]  Jerome A. Feldman,et al.  Modeling Embodied Lexical Development , 1997 .

[53]  M. Csíkszentmihályi Flow: The Classic Work on How to Achieve Happiness , 2002 .

[54]  A. Friedman Framing pictures: the role of knowledge in automatized encoding and memory for gist. , 1979, Journal of experimental psychology. General.

[55]  Bernard Suits,et al.  The Grasshopper: Games, Life and Utopia , 1978 .

[56]  F. Bartlett,et al.  Remembering: A Study in Experimental and Social Psychology , 1932 .

[57]  K. Groos,et al.  The play of animals : a study of animal life and instinct , 2022 .

[58]  S. Paradiso,et al.  Book Review: Affective Neuroscience: The Foundations of Human and Animal Emotions , 2000 .

[59]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[60]  Douglas Rushkoff,et al.  Open Source Democracy: How Online Communication is Changing Offline Politics , 2003 .

[61]  Ravin Balakrishnan,et al.  Keepin' it real: pushing the desktop metaphor with physics, piles and the pen , 2006, CHI.

[62]  S. Pinker The Stuff of Thought: Language as a Window into Human Nature , 2007 .

[63]  Edward E. Rigdon,et al.  Play, Flow, and the Online Search Experience , 2004 .

[64]  L. Manovich,et al.  The language of new media , 2001 .

[65]  Andrew B. King Website optimization - speed, search engine and conversion rate secrets , 2008 .

[66]  A. Bazin,et al.  What is cinema , 1967 .

[67]  S. Brookfield Becoming a Critically Reflective Teacher. Jossey-Bass Higher and Adult Education Series. , 1995 .

[68]  Cliff Lampe,et al.  A familiar face(book): profile elements as signals in an online social network , 2007, CHI.

[69]  Lynn Hasher,et al.  Is memory schematic , 1983 .

[70]  Heather Richter Lipford,et al.  Examining privacy and disclosure in a social networking community , 2007, SOUPS '07.

[71]  Terry Winograd,et al.  Understanding computers and cognition - a new foundation for design , 1987 .

[72]  Espen Aarseth Playing Research: Methodological approaches to game analysis , 2003 .

[73]  Alexander Alberro,et al.  Two-Way Mirror Power: Selected Writings by Dan Graham on His Art , 1999 .

[74]  Donald A. Norman,et al.  Some observations on mental models , 1987 .

[75]  K. Strater,et al.  Strategies and struggles with privacy in an online social networking community , 2008 .

[76]  L. Boroditsky Metaphoric structuring: understanding time through spatial metaphors , 2000, Cognition.

[77]  Kent Cartwright Shakespearean tragedy and its double , 1991 .

[78]  Hubert L. Dreyfus,et al.  Sense And Non-Sense , 1992 .

[79]  Gareth Monkman Greater than the sum of its parts , 2002 .

[80]  William McDonough,et al.  Cradle to Cradle: Remaking the Way We Make Things , 2002 .

[81]  Willett Kempton,et al.  Two Theories of Home Heat Control , 1986, Cogn. Sci..

[82]  Marvin Minsky,et al.  Semantic Information Processing , 1968 .

[83]  Ernest Edmonds New directions in interactive art collaboration , 2006 .

[84]  Todd C. Kelley,et al.  The Art of Innovation: Lessons in Creativity from IDEO, America's Leading Design Firm , 2001 .

[85]  Antti Oulasvirta,et al.  It's Mine, Don't Touch!: interactions at a large multi-touch display in a city centre , 2008, CHI.

[86]  Mihaly Csikszentmihalyi,et al.  Flow: the psychology of happiness , 1992 .

[87]  D. Meadows-Klue The Tipping Point: How Little Things Can Make a Big Difference , 2004 .

[88]  Chuck Palahniuk Stranger than Fiction: True Stories , 2004 .

[89]  E. Edmonds,et al.  Understanding the experience of interactive art: Iamascope in Beta_space , 2005 .

[90]  E. Edmonds,et al.  Living laboratories for interactive art , 2006 .

[91]  Jakob Nielsen,et al.  Homepage Usability: 50 Websites Deconstructed , 2001 .

[92]  Umberto Eco,et al.  A theory of semiotics , 1976, Advances in semiotics.

[93]  D. Norman Emotional design : why we love (or hate) everyday things , 2004 .

[94]  Steve Mulder,et al.  The User Is Always Right: A Practical Guide to Creating and Using Personas for the Web , 2006 .

[95]  T. Long Play: How it Shapes the Brain, Opens the Imagination, and Invigorates the Soul , 2009 .

[96]  Regine,et al.  We make money not art , 2011 .

[97]  Hector Rodriguez,et al.  The Playful and the Serious: An approximation to Huizinga's Homo Ludens , 2006, Game Stud..

[98]  C. Carver,et al.  Modeling: An analysis in terms of category accessibility. , 1983 .

[99]  Ben Weeks The Paradox of Choice: Why More Is Less , 2004 .

[100]  Charles Ashbacher,et al.  Triumph of the Nerds , 2002 .

[101]  Simone Mulargia,et al.  The Video Game Theory Reader 2 , 2011 .

[102]  Jeffrey Shaw,et al.  Future cinema : the cinematic imaginary after film , 2003 .

[103]  Ernest A. Edmonds,et al.  Approaches to interactive art systems , 2004, GRAPHITE '04.

[104]  D. Hindman The Virtual Community: Homesteading on the Electronic Frontier , 1996 .

[105]  Maurice Merleau-Ponty The World Of Perception , 2004 .

[106]  Tom Igoe Making Things Talk - Practical Methods for Connecting Physical Objects, Second Edition , 2011 .

[107]  Paul S. Breidenbach,et al.  The Medium of the Video Game , 2005 .

[108]  Maurice Merleau-Ponty The Primacy Of Perception , 1964 .

[109]  Bill Buxton,et al.  Multi-Touch Systems that I Have Known and Loved , 2009 .

[110]  Shani Bans,et al.  The Post Card: From Socrates to Freud and Beyond , 1980 .

[111]  Steven Johnson,et al.  Emergence: The Connected Lives of Ants, Brains, Cities, and Software , 2001 .

[112]  Jesper Juul,et al.  A Clash between Game and Narrative , 2011 .

[113]  Espen Aarseth,et al.  Computer Game Studies, Year One , 2001, Game Stud..

[114]  Donna L. Hoffman,et al.  Measuring the Customer Experience in Online Environments: A Structural Modeling Approach , 2000 .

[115]  M. Csíkszentmihályi Beyond boredom and anxiety , 1975 .

[116]  Marvin Minsky,et al.  A framework for representing knowledge , 1974 .

[117]  Espen Aarseth Cybertext: Perspectives on Ergodic Literature , 1997 .

[118]  Eric Paulos,et al.  The familiar stranger: anxiety, comfort, and play in public places , 2004, CHI.

[119]  Lucy Suchman Plans and situated actions: the problem of human-machine communication , 1987 .

[120]  R A Brooks,et al.  New Approaches to Robotics , 1991, Science.

[121]  Steven Poole,et al.  Trigger Happy: Videogames and the Entertainment Revolution , 2004 .

[122]  Adam N. Joinson,et al.  Looking at, looking up or keeping up with people?: motives and use of facebook , 2008, CHI.

[123]  Michael Schrage,et al.  Serious Play: How the World's Best Companies Simulate to Innovate , 1999 .

[124]  Rodney A. Brooks,et al.  Flesh and Machines: How Robots Will Change Us , 2002 .

[125]  Mary Deaton,et al.  The elements of user experience: user-centered design for the Web , 2003, INTR.

[126]  Alessandro Acquisti,et al.  Information revelation and privacy in online social networks , 2005, WPES '05.

[127]  J. Bargh,et al.  Automaticity in action: The unconscious as repository of chronic goals and motives. , 1996 .

[128]  Steven Johnson,et al.  Everything bad is good for you : how popular culture is making us smarter , 2006 .

[129]  Bill Moggridge,et al.  Designing interactions , 2006 .

[130]  Katie Salen,et al.  Rules of play: game design fundamentals , 2003 .

[131]  Roger C. Schank,et al.  Language and Memory , 1986, Cogn. Sci..

[132]  Celia Pearce,et al.  Theory Wars: An Argument Against Arguments in the so-called Ludology/Narratology Debate , 2005, DiGRA Conference.

[133]  Steven L. Kent The Ultimate History of Video Games , 2001 .

[134]  Arte Art Gallery of New South Wales , 2010 .

[135]  Tim O'Reilly,et al.  What is Web 2.0: Design Patterns and Business Models for the Next Generation of Software , 2007 .

[136]  Spike Cramphorn Blink: The Power of Thinking without Thinking / Strangers to Ourselves: Discovering the Adaptive Unconscious , 2006, Journal of Advertising Research.

[137]  Christiane Paul Whitney Challenges for a Ubiquitous Museum : Presenting and Preserving New Media , 2004 .

[138]  B. Sutton-Smith,et al.  The Ambiguity of Play , 2000 .

[139]  S. Field Screenplay: The Foundations of Screenwriting , 1979 .

[140]  M. Csíkszentmihályi Finding Flow: The Psychology of Engagement with Everyday Life , 1997 .

[141]  Jesper Juul Half-Real: Video Games between Real Rules and Fictional Worlds , 2005 .

[142]  W. Prinz A common-coding approach to perception and action , 1990 .

[143]  David Rokeby Transforming Mirrors: Subjectivity and Control in Interactive Media , 1995 .

[144]  G. Lakoff,et al.  Metaphors We Live by , 1981 .

[145]  Lera Boroditsky,et al.  The Roles of Body and Mind in Abstract Thought , 2002, Psychological science.

[146]  Trevor van Gorp,et al.  Design for Emotion and Flow , 2008 .

[147]  Steve Benford,et al.  Ambiguity as a resource for design , 2003, CHI '03.

[148]  chris-anderson The Long Tail: How Endless Choice is Creating Unlimited Demand , 2007 .

[149]  S. Freud Beyond the Pleasure Principle , 1925 .

[150]  Sadie Plant Zeros + ones : digital women + the new technoculture , 1997 .

[151]  Lucy Suchman,et al.  Understanding practice: video as a medium for reflection and design (excerpt) , 1992 .

[152]  D. Winnicott Playing and Reality , 1971 .

[153]  R. Caillois,et al.  Man, Play and Games , 1958 .

[154]  Rodney A. Brooks,et al.  Elephants don't play chess , 1990, Robotics Auton. Syst..

[155]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[156]  Gonzalo Frasca,et al.  Ludologists love stories, too: notes from a debate that never took place , 2003, DiGRA Conference.

[157]  Victor Fleming,et al.  オズの魔法使 = The wizard of Oz , 2005 .

[158]  Alex Wipperfurth,et al.  Brand Hijack: Marketing Without Marketing , 2005 .

[159]  J. Bargh,et al.  The psychology of action : linking cognition and motivation to behavior , 1999 .

[160]  Ea Edmonds,et al.  Living Laboratories: Making and Curating Interactive Art , 2006 .

[161]  Norbert Schwarz,et al.  Feelings and their motivational implications: Moods and the action sequence , 1996 .

[162]  Heloir,et al.  The Uncanny Valley , 2019, The Animation Studies Reader.