Decide or Delegate: How Script Knowledge Based Conversational Assistants Should Act in Inconclusive Situations

We present a study addressing the question how script knowledge based conversational assistants should act in situations of inconclusive information. Such situations occur for example in case of alternative or optional events that lead to multiple correct paths through the script. We have conducted a user study with four typical everyday activities (Making Coffee, Baking Cake, Finding the route to main station, Finding the route to camping ground) that may be represented in scripts. In this study, we have compared and evaluated four different presentation styles to handle situations of conflicting script information. A total of 182 persons participated in our study. The evaluation results show that, in case of a conflicting script state, users find the assistant most useful and are most satisfied if the assistant guesses the next correct event and provides a direct instruction instead of disclosing his incompetence. Alternatives in which the assistant delegates the decision to the user score worse.

[1]  Manfred Pinkal,et al.  Learning Script Knowledge with Web Experiments , 2010, ACL.

[2]  C. Welin.  Scripts, plans, goals and understanding, an inquiry into human knowledge structures: Roger C. Schank and Robert P. Abelson Hillsdale: Lawrence Erlbaum Associates, 1977. 248 pp. £ 10.60 hardcover , 1979 .

[3]  Ivan Titov,et al.  Inducing Neural Models of Script Knowledge , 2014, CoNLL.

[4]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[5]  Peter Clark,et al.  Learning Knowledge Graphs for Question Answering through Conversational Dialog , 2015, NAACL.

[6]  Raymond J. Mooney,et al.  Learning Statistical Scripts with LSTM Recurrent Neural Networks , 2016, AAAI.

[7]  Richard Edward Cullingford,et al.  Script application: computer understanding of newspaper stories. , 1977 .

[8]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[9]  Reid Swanson,et al.  Say Anything: A Massively Collaborative Open Domain Story Writing Companion , 2008, ICIDS.

[10]  D. A. Grant.  The latin square principle in the design and analysis of psychological experiments. , 1948, Psychological bulletin.

[11]  Lisa F. Rau,et al.  Information extraction and text summarization using linguistic knowledge acquisition , 1989, Inf. Process. Manag..

[12]  Manabu Okumura,et al.  Automatic Acquisition of Script Knowledge from a Text Collection , 2003, EACL.

[13]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[14]  Ivan Titov,et al.  A Hierarchical Bayesian Model for Unsupervised Induction of Script Knowledge , 2014, EACL.