When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose J UICER , a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed D IRECTOR model.

[1]  Edouard Grave,et al.  PEER: A Collaborative Language Model , 2022, ICLR.

[2]  J. Weston,et al.  Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls , 2022, ArXiv.

[3]  J. Weston,et al.  Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback , 2022, ACL.

[4]  Eric Michael Smith,et al.  BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage , 2022, ArXiv.

[5]  J. Weston,et al.  Director: Generator-Classifiers For Supervised Language Modeling , 2022, AACL.

[6]  Jeff Wu,et al.  Self-critiquing models for assisting human evaluators , 2022, ArXiv.

[7]  Jon Ander Campos,et al.  Training Language Models with Language Feedback , 2022, 2204.14146.

[8]  J. Weston,et al.  Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion , 2022, EMNLP.

[9]  Ryan J. Lowe,et al.  Training language models to follow instructions with human feedback , 2022, NeurIPS.

[10]  Niket Tandon,et al.  Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback , 2021, NAACL-HLT.

[11]  Jason Weston,et al.  Internet-Augmented Dialogue Generation , 2021, ACL.

[12]  Jason Weston,et al.  Beyond Goldfish Memory: Long-Term Open-Domain Conversation , 2021, ACL.

[13]  Dario Amodei,et al.  A General Language Assistant as a Laboratory for Alignment , 2021, ArXiv.

[14]  Matthew Richardson,et al.  NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction , 2021, NAACL.

[15]  Mohit Bansal,et al.  I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling , 2020, ACL.

[16]  Abigail See,et al.  Understanding and predicting user dissatisfaction in a neural generative chatbot , 2021, SIGDIAL.

[17]  Jason Weston,et al.  Deploying Lifelong Open-Domain Dialogue Learning , 2020, ArXiv.

[18]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[19]  Mary Williamson,et al.  Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills , 2020, ACL.

[20]  Jeremy Blackburn,et al.  The Pushshift Reddit Dataset , 2020, ICWSM.

[21]  Myle Ott,et al.  Unsupervised Cross-lingual Representation Learning at Scale , 2019, ACL.

[22]  Jason Weston,et al.  Learning from Dialogue after Deployment: Feed Yourself, Chatbot! , 2019, ACL.

[23]  Jason Weston,et al.  Learning through Dialogue Interactions by Asking Questions , 2016, ICLR.

[24]  Jason Weston,et al.  Dialogue Learning With Human-In-The-Loop , 2016, ICLR.