BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage

We present BlenderBot 3, a 175B parameter dialogue model capable of open-domain conversation with access to the internet and a long-term memory, and having been trained on a large number of user defined tasks. We release both the model weights and code, and have also deployed the model on a public web page to interact with organic users. This technical report describes how the model was built (architecture, model and training scheme), and details of its deployment, including safety mechanisms. Human evaluations show its superiority to existing open-domain dialogue agents, including its predecessors (Roller et al., 2021; Komeili et al., 2022). Finally, we detail our plan for continual learning using the data collected from deployment, which will also be publicly released. The goal of this research program is thus to enable the community to study ever-improving responsible agents that learn through interaction.

[1]  J. Weston,et al.  Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls , 2022, ArXiv.

[2]  J. Weston,et al.  Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback , 2022, ACL.

[3]  J. Weston,et al.  Director: Generator-Classifiers For Supervised Language Modeling , 2022, AACL.

[4]  Jeff Wu,et al.  Self-critiquing models for assisting human evaluators , 2022, ArXiv.

[5]  M. Shoeybi,et al.  Factuality Enhanced Language Models for Open-Ended Text Generation , 2022, ArXiv.

[6]  Eric Michael Smith,et al.  "I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset , 2022, ArXiv.

[7]  Xi Victoria Lin,et al.  OPT: Open Pre-trained Transformer Language Models , 2022, ArXiv.

[8]  Tom B. Brown,et al.  Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , 2022, ArXiv.

[9]  J. Weston,et al.  Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion , 2022, EMNLP.

[10]  Ryan J. Lowe,et al.  Training language models to follow instructions with human feedback , 2022, NeurIPS.

[11]  Geoffrey Irving,et al.  Red Teaming Language Models with Language Models , 2022, EMNLP.

[12]  Renelito Delos Santos,et al.  LaMDA: Language Models for Dialog Applications , 2022, ArXiv.

[13]  Y-Lan Boureau,et al.  SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures , 2021, ACL.

[14]  Jason Weston,et al.  Internet-Augmented Dialogue Generation , 2021, ACL.

[15]  Jason Weston,et al.  Beyond Goldfish Memory: Long-Term Open-Domain Conversation , 2021, ACL.

[16]  E. Cambria,et al.  Recent advances in deep learning based dialogue systems: a systematic survey , 2021, Artificial Intelligence Review.

[17]  Hwanjun Song,et al.  Learning From Noisy Labels With Deep Neural Networks: A Survey , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Shannon L. Spruit,et al.  SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems , 2022, ACL.

[19]  Jeff Wu,et al.  WebGPT: Browser-assisted question-answering with human feedback , 2021, ArXiv.

[20]  Po-Sen Huang,et al.  Scaling Language Models: Methods, Analysis & Insights from Training Gopher , 2021, ArXiv.

[21]  Jason Weston,et al.  Reason first, then respond: Modular Generation for Knowledge-infused Dialogue , 2021, EMNLP.

[22]  Jeffrey P. Bigham,et al.  A Survey of NLP-Related Crowdsourcing HITs: what works and what does not , 2021, ArXiv.

[23]  Nicholas Carlini,et al.  Unsolved Problems in ML Safety , 2021, ArXiv.

[24]  Adina Williams,et al.  Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models , 2021, ArXiv.

[25]  Michael S. Bernstein,et al.  On the Opportunities and Risks of Foundation Models , 2021, ArXiv.

[26]  Shannon L. Spruit,et al.  Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling , 2021, ArXiv.

[27]  Goran Glavas,et al.  RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models , 2021, ACL.

[28]  Nanyun Peng,et al.  Revealing Persona Biases in Dialogue Systems , 2021, ArXiv.

[29]  Jason Weston,et al.  Retrieval Augmentation Reduces Hallucination in Conversation , 2021, EMNLP.

[30]  Zhiyi Ma,et al.  Dynabench: Rethinking Benchmarking in NLP , 2021, NAACL.

[31]  Bo Liu,et al.  A Lifelong Learning Approach to Mobile Robot Navigation , 2021, IEEE Robotics and Automation Letters.

[32]  Naman Goyal,et al.  BASE Layers: Simplifying Training of Large, Sparse Models , 2021, ICML.

[33]  Namkee Park,et al.  Use of offensive language in human-artificial intelligence chatbot interaction: The effects of ethical ideology, social competence, and perceived humanlikeness , 2021, Comput. Hum. Behav..

[34]  Emily M. Bender,et al.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 , 2021, FAccT.

[35]  Charles Foster,et al.  The Pile: An 800GB Dataset of Diverse Text for Language Modeling , 2020, ArXiv.

[36]  Bing Liu,et al.  Continual Learning in Task-Oriented Dialogue Systems , 2020, EMNLP.

[37]  Edouard Grave,et al.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering , 2020, EACL.

[38]  Mary Williamson,et al.  Recipes for Building an Open-Domain Chatbot , 2020, EACL.

[39]  Joelle Pineau,et al.  Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program) , 2020, J. Mach. Learn. Res..

[40]  Jason Weston,et al.  Dialogue in the Wild: Learning from a Deployed Role-Playing Game with Humans and Bots , 2021, FINDINGS.

[41]  J. Weston,et al.  Recipes for Safety in Open-domain Chatbots , 2020, ArXiv.

[42]  Eric Michael Smith,et al.  Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions , 2020, ArXiv.

[43]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[44]  Jason Weston,et al.  Multi-Dimensional Gender Bias Classification , 2020, EMNLP.

[45]  Mary Williamson,et al.  Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills , 2020, ACL.

[46]  Agostino Poggi,et al.  A Survey on Troll Detection , 2020, Future Internet.

[47]  Quoc V. Le,et al.  Towards a Human-like Open-Domain Chatbot , 2020, ArXiv.

[48]  J. Weston,et al.  Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation , 2019, EMNLP.

[49]  J. Weston,et al.  The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents , 2019, ACL.

[50]  Myle Ott,et al.  Unsupervised Cross-lingual Representation Learning at Scale , 2019, ACL.

[51]  Jianfeng Gao,et al.  DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation , 2019, ACL.

[52]  Raghav Gupta,et al.  Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset , 2019, AAAI.

[53]  Jianfeng Gao,et al.  Challenges in Building Intelligent Open-domain Dialog Systems , 2019, ACM Trans. Inf. Syst..

[54]  Yejin Choi,et al.  The Curious Case of Neural Text Degeneration , 2019, ICLR.

[55]  Harry Shum,et al.  The Design and Implementation of XiaoIce, an Empathetic Social Chatbot , 2018, CL.

[56]  Dilek Z. Hakkani-Tür,et al.  Further Advances in Open Domain Dialog Systems in the Third Alexa Prize Socialbot Grand Challenge , 2020 .

[57]  Bill Byrne,et al.  Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset , 2019, EMNLP.

[58]  Jason Weston,et al.  Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack , 2019, EMNLP.

[59]  Ming-Wei Chang,et al.  Natural Questions: A Benchmark for Question Answering Research , 2019, TACL.

[60]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[61]  Andrew McCallum,et al.  Energy and Policy Considerations for Deep Learning in NLP , 2019, ACL.

[62]  Ming-Wei Chang,et al.  Latent Retrieval for Weakly Supervised Open Domain Question Answering , 2019, ACL.

[63]  Jason Weston,et al.  Learning to Speak and Act in a Fantasy Text Adventure Game , 2019, EMNLP.

[64]  Joelle Pineau,et al.  The Second Conversational Intelligence Challenge (ConvAI2) , 2019, The NeurIPS '18 Competition.

[65]  Thomas Wolf,et al.  TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents , 2019, ArXiv.

[66]  Susan T. Dumais,et al.  Improving Web Search Ranking by Incorporating User Behavior Information , 2019, SIGIR Forum.

[67]  Jason Weston,et al.  Learning from Dialogue after Deployment: Feed Yourself, Chatbot! , 2019, ACL.

[68]  Y-Lan Boureau,et al.  Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset , 2018, ACL.

[69]  Inioluwa Deborah Raji,et al.  Model Cards for Model Reporting , 2018, FAT.

[70]  Jason Weston,et al.  Wizard of Wikipedia: Knowledge-Powered Conversational agents , 2018, ICLR.

[71]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[72]  Sergey I. Nikolenko,et al.  Lost in Conversation: A Conversational Agent Based on the Transformer and Transfer Learning , 2019 .

[73]  Antoine Bordes,et al.  Training Millions of Personalized Dialogue Agents , 2018, EMNLP.

[74]  Zeyu Li,et al.  Learning Gender-Neutral Word Embeddings , 2018, EMNLP.

[75]  Ray Kurzweil,et al.  Learning Semantic Textual Similarity from Conversations , 2018, Rep4NLP@ACL.

[76]  Jason Weston,et al.  Personalizing Dialogue Agents: I have a dog, do you have pets too? , 2018, ACL.

[77]  Yi Pan,et al.  Conversational AI: The Science Behind the Alexa Prize , 2018, ArXiv.

[78]  Joelle Pineau,et al.  A Survey of Available Corpora for Building Data-Driven Dialogue Systems , 2015, Dialogue Discourse.

[79]  Jiliang Tang,et al.  A Survey on Dialogue Systems: Recent Advances and New Frontiers , 2017, SKDD.

[80]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[81]  Jason Weston,et al.  ParlAI: A Dialog Research Software Platform , 2017, EMNLP.

[82]  Eunsol Choi,et al.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , 2017, ACL.

[83]  Jason Weston,et al.  Learning Through Dialogue Interactions , 2016, ICLR.

[84]  Jason Weston,et al.  Dialogue Learning With Human-In-The-Loop , 2016, ICLR.

[85]  Lucas Dixon,et al.  Ex Machina: Personal Attacks Seen at Scale , 2016, WWW.

[86]  Ernest Davis,et al.  AI amusements: the tragic tale of Tay the chatbot , 2016, SIGAI.

[87]  Jianfeng Gao,et al.  A Human Generated MAchine Reading COmprehension Dataset , 2018 .

[88]  Preslav Nakov,et al.  Hunting for Troll Comments in News Community Forums , 2016, ACL.

[89]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[90]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[91]  Dumitru Erhan,et al.  Training Deep Neural Networks on Noisy Labels with Bootstrapping , 2014, ICLR.

[92]  Wiebke Wagner,et al.  Steven Bird, Ewan Klein and Edward Loper: Natural Language Processing with Python, Analyzing Text with the Natural Language Toolkit , 2010, Lang. Resour. Evaluation.

[93]  Estevam R. Hruschka,et al.  Toward an Architecture for Never-Ending Language Learning , 2010, AAAI.

[94]  N. Hara,et al.  Beyond vandalism: Wikipedia trolls , 2010, J. Inf. Sci..

[95]  Carl E. Rasmussen,et al.  The Need for Open Source Software in Machine Learning , 2007, J. Mach. Learn. Res..