Be Brief, And They Shall Learn: Generating Concise Language Feedback for a Computer Tutor

To investigate whether more concise Natural Language feedback improves learning, we developed two Natural Language generators (DIAG-NLP1 and DIAG-NLP2), to provide feedback in an Intelligent Tutoring System that teaches troubleshooting. We systematically evaluated them in a three way comparison that included the original system, which generates overly repetitive feedback. We found that DIAG-NLP2, the generator which intuitively produces the best, corpus-based language, does engender the most learning. Distinguishing features of the more effective feedback are: it obeys Grice's maxim of brevity, it is more directive and uses a specific type of referring expressions. Interestingly, simpler ways of restructuring the original repetitive feedback as done in DIAG-NLP1, such as exploiting the hierarchical structure of the domain, were not effective. Since the design of interfaces to Intelligent Tutoring Systems often includes verbal feedback, we suggest that: if the number of different contexts in which verbal feedback is provided is high, such feedback should be based on corpus studies, and generated by techniques more sophisticated than template filling.

[1]  Barbara Di Eugenio,et al.  Minimal Text Structuring to Improve the Generation of Feedback in Intelligent Tutoring Systems , 2003, FLAIRS Conference.

[2]  Arthur C. Graesser,et al.  When Are Tutorial Dialogues More Effective Than Reading? , 2007, Cogn. Sci..

[3]  Trude Heift,et al.  Error-specific and individualised feedback in a Web-based language tutoring system: Do they read it? , 2001, ReCALL.

[4]  James Shaw.  A Corpus-based Analysis for the Ordering of Clause Aggregation Operators , 2002, COLING.

[5]  Michael Pressley,et al.  Scaffolding student learning: Instructional approaches and issues. , 1997 .

[6]  Lael J. Schooler,et al.  The Disruptive Potential of Immediate Feedback , 2008 .

[7]  J. Hawkins.  Definiteness and indefiniteness: a study in reference and grammaticality prediction , 1978 .

[8]  Davide Fossati,et al.  Learning Linked Lists: Experiments with the iList System , 2008, Intelligent Tutoring Systems.

[9]  Vincent Aleven,et al.  A Tutorial Dialog System to Support Self- Explanation: Evaluation and Open Questions , 2003 .

[10]  Jaime R. Carbonell,et al.  AI in CAI : An artificial intelligence approach to computer-assisted instruction , 1970 .

[11]  Michael Drake,et al.  Scaffolding techniques of expert human tutors. , 1997 .

[12]  M. Reape,et al.  Just what is aggregation anyway ? , 2007 .

[13]  Xiaorong Huang,et al.  Paraphrasing and Aggregating Argumentative Texts Using Text Structure , 1996, INLG.

[14]  Donia Scott,et al.  Document Structure , 2003, CL.

[15]  Heather H. Mitchell,et al.  AutoTutor: A tutor with dialogue in natural language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[16]  Brady Clark,et al.  Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems , 2006, Int. J. Artif. Intell. Educ..

[17]  Jean McKendree,et al.  Effective Feedback Content for Tutoring Complex Skills , 1990, Hum. Comput. Interact..

[18]  Siobhan Chapman.  Logic and Conversation , 2005 .

[19]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[20]  Kenneth R. Koedinger,et al.  Can a Polite Intelligent Tutoring System Lead to Improved Learning Outside of the Lab? , 2007, AIED.

[21]  J. Sinclair,et al.  Towards an Analysis of Discourse: The English Used by Teachers and Pupils , 1975 .

[22]  A. Graesser,et al.  LEARNING WHILE HOLDING A CONVERSATION WITH A COMPUTER , 2005 .

[23]  Davide Fossati,et al.  Beyond the code-and-count analysis of tutoring dialogues , 2007, AIED.

[24]  Davide Fossati,et al.  Aggregation Improves Learning: Experiments in Natural Language Generation for Intelligent Tutoring Systems , 2005, ACL.

[25]  Claus Zinn,et al.  Intelligent Information Presentation for Tutoring Systems , 2005 .

[26]  Douglas M. Towne.  Approximate Reasoning Techniques for Intelligent Diagnostic Instruction , 1997 .

[27]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[28]  Carolyn Penstein Rosé,et al.  Spoken Versus Typed Human and Computer Dialogue Tutoring , 2006, Int. J. Artif. Intell. Educ..

[29]  Carolyn Penstein Rosé,et al.  Evaluating the Effectiveness of Tutorial Dialogue Instruction in an Exploratory Learning Context , 2006, Intelligent Tutoring Systems.

[30]  Barbara Di Eugenio,et al.  Generating Natural Language Aggregations Using a Propositional Representation of Sets , 2002, FLAIRS Conference.

[31]  Ryan Shaun Joazeiro de Baker,et al.  Adapting to When Students Game an Intelligent Tutoring System , 2006, Intelligent Tutoring Systems.

[32]  Ehud Reiter,et al.  Book Reviews: Building Natural Language Generation Systems , 2000, CL.

[33]  I. Arroyo,et al.  Addressing Cognitive Differences and Gender During Problem Solving , 2006 .

[34]  Barbara A. Fox.  The Human Tutorial Dialogue Project: Issues in the Design of instructional Systems , 1993 .

[35]  Pamela W. Jordan,et al.  Intentional influences on object redescriptions in dialogue: evidence from an empirical study , 2000 .

[36]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[37]  Claus Zinn,et al.  Generating Tutorial Feedback with Affect , 2004, FLAIRS.

[38]  Cécile Paris,et al.  Tailoring Object Descriptions to a User's Level of Expertise , 1988, Comput. Linguistics.

[39]  H. H. Clark.  Arenas of language use , 1993 .

[40]  Ning Wang,et al.  The politeness effect: Pedagogical agents and learning outcomes , 2008, Int. J. Hum. Comput. Stud..

[41]  Barbara Di Eugenio,et al.  The DIAG experiments: Natural Language Generation for Intelligent Tutoring Systems , 2002, INLG.

[42]  Davide Fossati,et al.  Natural Language Generation for Intelligent Tutoring Systems: a case study , 2005, AIED.

[43]  Yuh-Fang Chang.  On the use of the immediate recall task as a measure of second language reading comprehension , 2006 .

[44]  Joseph P. Magliano,et al.  Collaborative dialogue patterns in naturalistic one-to-one tutoring , 1995 .

[45]  Takashi Yamauchi,et al.  Learning from human tutoring , 2001, Cogn. Sci..

[46]  Benoit Lavoie,et al.  A Fast and Portable Realizer for Text Generation Systems , 1997, ANLP.

[47]  Rong Jin,et al.  Linguistic theories in efficient multimodal reference resolution: an empirical investigation , 2005, IUI.

[48]  R. Moreno,et al.  Does the modality principle hold for different media? A test of the method-affects-learning hypothesis , 2006, J. Comput. Assist. Learn..

[49]  V. Aleven,et al.  Help Seeking and Help Design in Interactive Learning Environments , 2003 .

[50]  Penelope Brown,et al.  Politeness: Some Universals in Language Usage , 1989 .

[51]  Michael White,et al.  EXEMPLARS: A Practical, Extensible Framework For Dynamic Text Generation , 1998, INLG.

[52]  R R Burton,et al.  Toward a natural-language capability for computer-assisted instruction , 1986 .

[53]  Karen A. Schriver.  Dynamics in document design , 1998 .

[54]  C M Cullum,et al.  Brief recall tasks and memory assessment in Alzheimer's disease. , 1999, Applied neuropsychology.

[55]  Karen A. Schriver.  Dynamics in Document Design: Creating Text for Readers , 1996 .

[56]  Xin Lu,et al.  Expert Tutoring and Natural Language Feedback in Intelligent Tutoring Systems , 2006, ICCE.

[57]  G. L. Riley,et al.  A comparison of recall and summary protocols as measures of second language reading comprehension , 1996 .

[58]  Susan E. Gathercole,et al.  The Role of Sentence Recall in Reading and Language Skills of Children with Learning Difficulties. , 2005 .

[59]  Stellan Ohlsson,et al.  Expert vs. Non-expert Tutoring: Dialogue Moves, Interaction Patterns and Multi-utterance Turns , 2009, CICLing.

[60]  Andrew Kehler,et al.  Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction , 2000, AAAI/IAAI.

[61]  Barbara Di Eugenio,et al.  MUP - The UIC Standoff Markup Tool , 2002, SIGDIAL Workshop.

[62]  Vincent Aleven,et al.  An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor , 2002, Cogn. Sci..

[63]  Michelene T. H. Chi,et al.  Self-Explanations: How Students Study and Use Examples in Learning To Solve Problems. Technical Report No. 9. , 1987 .

[64]  Joel A. Michael,et al.  Hinting as a Tactic in One-on-One Tutoring , 1996 .

[65]  Stuart C. Shapiro.  SNePS: a logic for natural language understanding and commonsense reasoning , 2000 .

[66]  John N. Williams,et al.  Memory, Attention, and Inductive Learning , 1999, Studies in Second Language Acquisition.