Towards a crowdsourced solution for the authoring bottleneck in interactive narratives

Interactive Storytelling research has produced a wealth of technologies that can be employed to create personalised narrative experiences, in which the audience takes a participating rather than observing role. But so far this technology has not led to the production of large scale playable interactive story experiences that realise the ambitions of the field. One main reason for this state of affairs is the difficulty of authoring interactive stories, a task that requires describing a huge amount of story building blocks in a machine friendly fashion. This is not only technically and conceptually more challenging than traditional narrative authoring but also a scalability problem. This thesis examines the authoring bottleneck through a case study and a literature survey and advocates a solution based on crowdsourcing. Prior work has already shown that combining a large number of example stories collected from crowd workers with a system that merges these contributions into a single interactive story can be an effective way to reduce the authorial burden. As a refinement of such an approach, this thesis introduces the novel concept of Crowd Task Adaptation. It argues that in order to maximise the usefulness of the collected stories, a system should dynamically and intelligently analyse the corpus of collected stories and based on this analysis modify the tasks handed out to crowd workers. Two authoring systems, ENIGMA and CROSCAT, which show two radically different approaches of using the Crowd Task Adaptation paradigm have been implemented and are described in this thesis. While ENIGMA adapts tasks through a realtime dialog between crowd workers and the system that is based on what has been learned from previously collected stories, CROSCAT modifies the backstory given to crowd workers in order to optimise the distribution of branching points in the tree structure that combines all collected stories. Two experimental studies of crowdsourced authoring are also presented. They lead to guidelines on how to employ crowdsourced authoring effectively, but more importantly the results of one of the studies demonstrate the effectiveness of the Crowd Task Adaptation approach.

[1]  Mark O. Riedl,et al.  WeQuest: scalable alternate reality games through end-user content authoring , 2011, Advances in Computer Entertainment Technology.

[2]  Michael Mateas,et al.  Comme il Faut 2: a fully realized model for socially-oriented gameplay , 2010, FDG.

[3]  Stéphane Donikian,et al.  Writing Interactive Fiction Scenarii with DraMachina , 2004, TIDSE.

[4]  Assumptions underlying the Erasmatron interactive storytelling engine ChrisCrawford , 1999 .

[5]  Michael Lebowitz,et al.  Story-telling as planning and learning , 1985 .

[6]  Roger B. Dannenberg,et al.  TagATune: A Game for Music and Sound Annotation , 2007, ISMIR.

[7]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[8]  Michael Mateas,et al.  Novice-Friendly Authoring of Plan-Based Interactive Storyboards , 2010, AIIDE.

[9]  Szilas Nicolas,et al.  Structural Models for Interactive Drama , 2002 .

[10]  Stacy Marsella,et al.  PsychSim: Modeling Theory of Mind with Decision-Theoretic Agents , 2005, IJCAI.

[11]  Michael Mateas,et al.  Authoring Game-based Interactive Narrative using Social Games and Comme il , 2010 .

[12]  A. Swiderska,et al.  eCute: Difference is good , 2012 .

[13]  David V. Pynadath,et al.  PsychSim: Agent-based Modeling of Social Interactions and Influence , 2004, ICCM.

[14]  Manuel Blum,et al.  Verbosity: a game for collecting common-sense facts , 2006, CHI.

[15]  Chris Crawford Deikto: an application of the weak sapir-whorf hypothesis , 2008, Creating '08.

[16]  Mei Si,et al.  Modeling Rich Characters in Interactive Narrative Games , 2010 .

[17]  Ana Paiva,et al.  eCIRCUS: building voices for autonomous speaking agents , 2007, SSW.

[18]  Ana Paiva,et al.  Using rituals to express cultural differences in synthetic characters , 2009, AAMAS.

[19]  Henry Lieberman,et al.  Watch what I do: programming by demonstration , 1993 .

[20]  Leonidas J. Guibas,et al.  A dichromatic framework for balanced trees , 1978, 19th Annual Symposium on Foundations of Computer Science (sfcs 1978).

[21]  Ana Paiva,et al.  Improving Adaptiveness in Autonomous Characters , 2008, IVA.

[22]  Olivier Balet INSCAPE An Authoring Platform for Interactive Storytelling , 2007, International Conference on Virtual Storytelling.

[23]  Marc Cavazza,et al.  Developing re-usable interactive storytelling technologies , 2004, IFIP Congress Topical Sessions.

[24]  Ana Paiva,et al.  FearNot! - An Experiment in Emergent Narrative , 2005, IVA.

[25]  Magy Seif El-Nasr,et al.  Interaction, Narrative, and Drama Creating an Adaptive Interactive Narrative using Performance Arts Theories , 2007 .

[26]  Dirk Heylen,et al.  The Virtual Storyteller: story creation by intelligent agents , 2003 .

[27]  J. Murray Hamlet on the Holodeck , 1997 .

[28]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[29]  Richard Fikes,et al.  STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving , 1971, IJCAI.

[30]  Daniel Kudenko,et al.  Schemas in Directed Emergent Drama , 2008, ICIDS.

[31]  Hartmut Koenitz,et al.  Genres, Structures and Strategies in Interactive Digital Narratives - Analyzing a Body of Works Created in ASAPS , 2012, ICIDS.

[32]  Ivo Swartjes,et al.  Iterative Authoring Using Story Generation Feedback: Debugging or Co-creation? , 2009, ICIDS.

[33]  Marc Cavazza,et al.  Interactive Storytelling with Literary Feelings , 2007, ACII.

[34]  Andrew Stern,et al.  Façade: An Experiment in Building a Fully-Realized Interactive Drama , 2003 .

[35]  Ulrike Spierling Adding Aspects of "Implicit Creation" to the Authoring Process in Interactive Storytelling , 2007, International Conference on Virtual Storytelling.

[36]  Ana Paiva,et al.  Feeling and Reasoning: A Computational Model for Emotional Characters , 2005, EPIA.

[37]  Andrew Stern,et al.  Embracing the Combinatorial Explosion: A Brief Prescription for Interactive Story R&D , 2008, ICIDS.

[38]  Ian D. Watson,et al.  An Introduction to Case-Based Reasoning , 1995, UK Workshop on Case-Based Reasoning.

[39]  Boyang Li,et al.  Toward Autonomous Crowd-Powered Creation of Interactive Narratives , 2012, INT@AIIDE.

[40]  Vadim Bulitko,et al.  Interactive Narrative: An Intelligent Systems Approach , 2012, AI Mag..

[41]  David Thue,et al.  Interactive Storytelling: A Player Modelling Approach , 2007, AIIDE.

[42]  Ana Paiva,et al.  An Affectively Driven Planner for Synthetic Characters , 2006, ICAPS.

[43]  Michael Mateas,et al.  Wide Ruled: A Friendly Interface to Author-Goal Based Story Generation , 2007, International Conference on Virtual Storytelling.

[44]  Vladimir Propp,et al.  Morphology of the folktale , 1959 .

[45]  H. Lieberman Common Consensus : a web-based game for collecting commonsense goals , 2007 .

[46]  Judy Robertson,et al.  Adventure Author: a learning environment to support creative design , 2007, IDC.

[47]  Wolfgang Nejdl,et al.  Extracting Semantics Relationships between Wikipedia Categories , 2006, SemWiki.

[48]  Robert Michael Young,et al.  Managing interaction between users and agents in a multi-agent storytelling environment , 2003, AAMAS '03.

[49]  Nicolas Szilas,et al.  Workshop on Authoring Methods and Conception in Interactive Storytelling , 2009, ICIDS.

[50]  Daniel Kudenko,et al.  Dynamic Generation of Dilemma-based Interactive Narratives , 2007, AIIDE.

[51]  Ana Paiva,et al.  Agents with Emotional Intelligence for Storytelling , 2011, ACII.

[52]  Ana Paiva,et al.  Emergent Stories Facilitated , 2008, ICIDS.

[53]  Roger C. Schank,et al.  SCRIPTS, PLANS, GOALS, AND UNDERSTANDING , 1988 .

[54]  Ruth Aylett,et al.  Double Appraisal for Synthetic Characters , 2007, IVA.

[55]  Ruth Aylett,et al.  Purposeful Authoring for Emergent Narrative , 2008, ICIDS.

[56]  Alex Mitchell,et al.  Reading Again for the First Time: A Model of Rereading in Interactive Stories , 2012, ICIDS.

[57]  Sue Thomas,et al.  A Million Penguins Research Report , 2008 .

[58]  Joseph Weizenbaum,et al.  ELIZA—a computer program for the study of natural language communication between man and machine , 1966, CACM.

[59]  Sarit Kraus,et al.  Social Narrative Adaptation using Crowdsourcing , 2013, CMN.

[60]  Brian Magerko A Proposal for an Interactive Drama Architecture , 2002 .

[61]  Robert Michael Young,et al.  A Use of Flashback and Foreshadowing for Surprise Arousal in Narrative Using a Plan-Based Approach , 2008, ICIDS.

[62]  Ana Paiva,et al.  So tell me what happened: turning agent-based interactive drama into comics , 2008, AAMAS.

[63]  Caitlin Kelleher Supporting Storytelling in a Programming Environment for Middle School Children , 2009, ICIDS.

[64]  Manuel Blum,et al.  Peekaboom: a game for locating objects in images , 2006, CHI.

[65]  Marc Cavazza,et al.  Generation of humorous situations in cartoons through plan-based formalisations , 2003 .

[66]  Blai Bonet,et al.  Planning as heuristic search , 2001, Artif. Intell..

[67]  S. Folkman,et al.  Stress, appraisal, and coping , 1974 .

[68]  Ivo Swartjes,et al.  Let's Pretend I Had a Sword: Late Commitment in Emergent Narrative , 2008, ICIDS.

[69]  Andrew Stern,et al.  Structuring Content in the Façade Interactive Drama Architecture , 2005, AIIDE.

[70]  Boyang Li Narrative Intelligence Without (Domain) Boundaries , 2012 .

[71]  Ivo Swartjes Using Narrative Cases to Author Interactive Story Content , 2007, ICEC.

[72]  Graham J Hole,et al.  How to Design and Report Experiments , 2002 .

[73]  Marc Cavazza,et al.  From Debugging to Authoring: Adapting Productivity Tools to Narrative Content Description , 2008, ICIDS.

[74]  Stephen G. Ware,et al.  Foreseeing Meaningful Choices , 2014, AIIDE.

[75]  Nicolas Szilas,et al.  Authoring Issues beyond Tools , 2009, ICIDS.

[76]  Kim Binsted,et al.  The Limitations of a Propp-based Approach to Interactive Drama , 2007, AAAI Fall Symposium: Intelligent Narrative Technologies.

[77]  R. Michael Young,et al.  An Overview of the Mimesis Architecture: Integrating Intelligent Narrative Control into an Existing Gaming Environment , 2001 .

[78]  Carlo Strapparava,et al.  An Authoring Tool for Intelligent Educational Games , 2001, International Conference on Virtual Storytelling.

[79]  Bradley C. Love,et al.  CAB: Connectionist Analogy Builder , 2003, Cogn. Sci..

[80]  Dave Beck,et al.  Theatre of the Oppressed , 2010 .

[81]  Y. Benkler,et al.  Commons‐based Peer Production and Virtue* , 2006 .

[82]  William Thompson Price,et al.  The Technique of the Drama , 2008 .

[83]  Mei Si Social Norms Models in Thespian : Using Decision Theoretical Framework for Interactive Dramas , .

[84]  Jeff Orkin Using Online Games to Capture, Generate, and Understand Natural Language [Invited Talk] , 2011, ENLG.

[85]  Ruth Aylett,et al.  Unscripted narrative for affectively driven characters , 2005, IEEE Computer Graphics and Applications.

[86]  Josep Blat,et al.  Writing Consistent Stories based on Structured Multi-Authored Narrative Spaces , 2013, CMN.

[87]  Michael Kriegel,et al.  Authoring Emergent Narrative-based Games , 2008, J. Game Dev..

[88]  Hartmut Koenitz Extensible Tools for Practical Experiments in IDN: The Advanced Stories Authoring and Presentation System , 2011, ICIDS.

[89]  Joseph Campbell,et al.  The Hero with a Thousand Faces , 1949 .

[90]  Jeff Orkin,et al.  Semi-automatic task recognition for interactive narratives with EAT & RUN , 2010, FDG.

[91]  M. Haahr,et al.  Interactive Storytelling , 2012, Lecture Notes in Computer Science.

[92]  Georges Polti The Thirty-Six Dramatic Situations , 1916 .

[93]  Hartmut Koenitz An Iterative Approach towards Interactive Digital Narrative - Early Results with the Advanced Stories Authoring and Presentation System , 2012, ICWL Workshops.

[94]  Stacy Marsella,et al.  Integrating Story-Centric and Character-Centric Processes for Authoring Interactive Drama , 2008, AIIDE.

[95]  R. Aylett Narrative in Virtual Environments - Towards Emergent Narrative , 1999 .

[96]  Marc Cavazza,et al.  Character-Based Interactive Storytelling , 2002, IEEE Intell. Syst..

[97]  D. Kudenko,et al.  A User Model for the Generation of Dilemma-based Interactive Narratives , 2007 .

[98]  Ido Iurgel,et al.  Cyranus - An Authoring Tool for Interactive Edutainment Applications , 2006, Edutainment.

[99]  Stacy Marsella,et al.  Thespian: using multi-agent fitting to craft interactive drama , 2005, AAMAS '05.

[100]  Boyang Li,et al.  Data-Driven Alibi Story Telling for Social Believability , 2014 .

[101]  Reid Swanson,et al.  A Data-Driven Case-Based Reasoning Approach to Interactive Storytelling , 2010, ICIDS.

[102]  Huaxin Wei Structuring Narrative Interaction: What We Can Learn from Heavy Rain , 2011, ICIDS.

[103]  Kim B. Clark,et al.  The Architecture of Participation: Does Code Architecture Mitigate Free Riding in the Open Source Development Model? , 2006, Manag. Sci..

[104]  Andrew Stern,et al.  Failing Believably: Toward Drama Management with Autonomous Actors in Interactive Narratives , 2006, TIDSE.

[105]  Ralf Engel SPIN : A Semantic Parser for Spoken Dialog Systems , 2006 .

[106]  C. Crawford Chris Crawford on Interactive Storytelling , 2004 .

[107]  Ana Paiva,et al.  "I Want to Slay That Dragon!" - Influencing Choice in Interactive Storytelling , 2010, ICIDS.

[108]  Ivo Swartjes Whose story is it anyway? How improv informs agency and authorship of emergent narrative , 2010 .

[109]  Reid Swanson,et al.  Say Anything: A Massively Collaborative Open Domain Story Writing Companion , 2008, ICIDS.

[110]  K. Dautenhahn,et al.  Inter-cultural differences in response to a computer-based anti-bullying intervention , 2010 .

[111]  Mark O. Riedl,et al.  Narrative generation: balancing plot and character , 2004 .

[112]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[113]  Richard Evans,et al.  Versu—A Simulationist Storytelling System , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[114]  Ruth Aylett,et al.  Distributed Drama Management: Beyond Double Appraisal in Emergent Narrative , 2012, ICIDS.

[115]  Ruth Aylett,et al.  Crowd-Sourced AI Authoring with ENIGMA , 2010, ICIDS.

[116]  Rudolf Kadlec,et al.  Emohawk: Searching for a "Good" Emergent Narrative , 2009, ICIDS.

[117]  Steve Hoffmann,et al.  How Authors Benefit from Linear Logic in the Authoring Process of Interactive Storyworlds , 2011, ICIDS.

[118]  C. Brémond Logique du récit , 1973 .

[119]  Mark O. Riedl Incorporating Authorial Intent into Generative Narrative Systems , 2009, AAAI Spring Symposium: Intelligent Narrative Technologies II.

[120]  Hans-Georg Struck Telling Stories Knowing Nothing: Tackling the Lack of Common Sense Knowledge in Story Generation Systems , 2005, International Conference on Virtual Storytelling.

[121]  Douglas B. Lenat,et al.  CYC: a large-scale investment in knowledge infrastructure , 1995, CACM.

[122]  S. Turner Minstrel: a computer model of creativity and storytelling , 1993 .

[123]  Reid Swanson,et al.  A Comparison of Retrieval Models for Open Domain Story Generation , 2009, AAAI Spring Symposium: Intelligent Narrative Technologies II.

[124]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[125]  Yijie Wang,et al.  Storytelling with Adjustable Narrator Styles and Sentiments , 2014, ICIDS.

[126]  Brian Magerko,et al.  Scribe: A Tool for Authoring Event Driven Interactive Drama , 2006, TIDSE.

[127]  Matthias Stifter,et al.  U-Create: Creative Authoring Tools for Edutainment Applications , 2006, TIDSE.

[128]  Ana Paiva,et al.  Virtual learning intervention to reduce bullying victimization in primary school: a controlled trial. , 2010, Journal of child psychology and psychiatry, and allied disciplines.

[129]  Ido Iurgel,et al.  Workshop and Panel: The Authoring Process in Interactive Storytelling , 2008, ICIDS.

[130]  James M. Thomas Author in the Loop : Using Mixed-Initiative Planning to Improve Interactive Narrative , 2006 .

[131]  Ana Paiva,et al.  An Authoring Tool for an Emergent Narrative Storytelling System , 2007, AAAI Fall Symposium: Intelligent Narrative Technologies.

[132]  Jeff Orkin,et al.  The Restaurant Game: Learning Social Behavior and Language from Thousands of Players Online , 2008, J. Game Dev..

[133]  Carlos Delgado-Mata,et al.  Managing emergent character-based narrative , 2008, INTETAIN '08.

[134]  Jeff Orkin,et al.  Automatic learning and generation of social behavior from collective human gameplay , 2009, AAMAS.

[135]  Marilyn A. Walker,et al.  All the World's a Stage: Learning Character Models from Film , 2011, AIIDE.

[136]  Paolo Petta,et al.  Backstory Authoring for Affective Agents , 2012, ICIDS.

[137]  Michael Mateas,et al.  Social Story Worlds With Comme il Faut , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[138]  Daniel Kudenko,et al.  Duality of Actor and Character Goals in Virtual Drama , 2009, IVA.

[139]  Push Singh,et al.  The Open Mind Common Sense project , 2002 .

[140]  Ruth Aylett,et al.  ORIENT: An Inter-cultural role-play game , 2008 .

[141]  Dieter Grasbon,et al.  a morphological approach to interactive storytelling , 2001 .

[142]  Ana Paiva,et al.  FearNot! - An Emergent Narrative Approach to Virtual Dramas for Anti-bullying Education , 2007, International Conference on Virtual Storytelling.

[143]  Mark O. Riedl,et al.  Story Planning with Vignettes: Toward Overcoming the Content Production Bottleneck , 2008, ICIDS.

[144]  Mariët Theune,et al.  The Virtual Storyteller : story generation by simulation , 2008 .

[145]  AnHai Doan,et al.  Mass Collaboration Systems on the World-Wide Web , 2010 .

[146]  Chris Fairclough,et al.  Story Games and the OPIATE System: Using Case-Based Planning for Structuring Plots with an Expert Story Director Agent and Enacting them in a Socially Simulated Game World , 2004 .

[147]  Brenda Laurel,et al.  Computers as theatre , 1991 .

[148]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[149]  Frank Wannemaker Motif Index Of Folk Literature , 2016 .

[150]  Santiago Ontañón,et al.  Textual vs. Graphical Interaction in an Interactive Fiction Game , 2010, ICIDS.

[151]  Nicolas Szilas,et al.  IDtension: a narrative engine for Interactive Drama , 2003 .

[152]  G. A. Miller THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[153]  Robert Michael Young,et al.  Narrative Planning: Balancing Plot and Character , 2010, J. Artif. Intell. Res..

[154]  Stacy Marsella,et al.  Directorial Control in a Decision-Theoretic Framework for Interactive Narrative , 2009, ICIDS.

[155]  R. Michael Young,et al.  An architecture for integrating plan-based behavior generation with interactive game environments , 2004, J. Game Dev..

[156]  Michael Mateas,et al.  Interactive Story Generation for Writers: Lessons Learned from the Wide Ruled Authoring Tool , 2009 .

[157]  Stephen G. Ware,et al.  Achieving the Illusion of Agency , 2012, ICIDS.

[158]  Anand S. Rao,et al.  BDI Agents: From Theory to Practice , 1995, ICMAS.

[159]  Ruth Aylett,et al.  Emergent Narrative as a Novel Framework for Massively Collaborative Authoring , 2008, IVA.

[160]  Raquel Hervás,et al.  Story plot generation based on CBR , 2004, Knowl. Based Syst..

[161]  Wolfgang Müller,et al.  Scenejo - An Interactive Storytelling Platform , 2005, International Conference on Virtual Storytelling.