FROM DATA TO DESIGN

Spoken dialog systems have been a holy grail of computer science since Turing devised his test for intelligence, and an icon of science fiction from well before that. Although there are machines you can talk to – interactive voice response (IVR) systems which answer phones, voice command systems in cars, and the Furby toy come to mind – holding a conversation with a machine, spoken or typed, is still highly problematic. It seems there is something fundamental we don't understand about the way humans use language. One popular approach is to ignore our ignorance and assume that machines can figure it out for themselves using machine learning or some form of statistical modeling of a corpus of text. Corpus analysis, however, like archeology, attempts to understand human action by looking at what we leave behind. Instead we take a software agents approach and model the language production process. The SERA project is our latest effort looking at what other disciplines can tell us about language use in context. We find that, naturally, published work focuses on the interesting while the engineering challenge is to capture the essential but often mundane. This paper proposes a narrative approach based on Vygotsky's view of psychology that captures “the big picture”. The paper finishes with an outline for a tool that merges the functionality of more conventional annotation tools with that of existing scripting environments for conversational agents.