PLACES: Prompting Language Models for Social Conversation Synthesis

Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversation excerpts sampled from a human-collected multi-party dataset.

[1]  Ronan Le Bras,et al.  SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization , 2022, ArXiv.

[2]  Dilek Z. Hakkani-Tür,et al.  Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding , 2022, ArXiv.

[3]  Amir Saffari,et al.  CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing , 2022, AACL.

[4]  Yannick Versley,et al.  LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging , 2022, COLING.

[5]  A. Kalai,et al.  Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies , 2022, ICML.

[6]  J. Weston,et al.  Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback , 2022, ACL.

[7]  Eric Michael Smith,et al.  BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage , 2022, ArXiv.

[8]  Y. Altun,et al.  LAD: Language Models as Data for Zero-Shot Dialog , 2022, SIGDIAL.

[9]  Bill Dolan,et al.  GODEL: Large-Scale Pre-Training for Goal-Directed Dialog , 2022, ArXiv.

[10]  Xi Victoria Lin,et al.  OPT: Open Pre-trained Transformer Language Models , 2022, ArXiv.

[11]  Sang-Woo Lee,et al.  Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models , 2022, NAACL.

[12]  Pau Rodríguez López,et al.  Data Augmentation for Intent Classification with Off-the-shelf Large Language Models , 2022, NLP4CONVAI.

[13]  M. Lewis,et al.  Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? , 2022, Conference on Empirical Methods in Natural Language Processing.

[14]  Yufei Wang,et al.  PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks , 2022, ACL.

[15]  Yu Zhang,et al.  Generating Training Data with Language Models: Towards Zero-Shot Language Understanding , 2022, NeurIPS.

[16]  Kyunghyun Cho,et al.  LINDA: Unsupervised Learning to Interpolate in Natural Language Processing , 2021, ArXiv.

[17]  M. Lewis,et al.  MetaICL: Learning to Learn In Context , 2021, NAACL.

[18]  Pascale Fung,et al.  Few-Shot Bot: Prompt-Based Learning for Dialogue Systems , 2021, ArXiv.

[19]  Dilek Z. Hakkani-Tür,et al.  Generative Conversational Networks , 2021, SIGDIAL.

[20]  Gunhee Kim,et al.  How Robust are Fact Checking Systems on Colloquial Claims? , 2021, NAACL.

[21]  Zhou Yu,et al.  LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing , 2021, ACL.

[22]  S. Riedel,et al.  Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity , 2021, ACL.

[23]  Ondvrej Duvsek,et al.  AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models , 2021, NLP4CONVAI.

[24]  Charles Foster,et al.  The Pile: An 800GB Dataset of Diverse Text for Language Modeling , 2020, ArXiv.

[25]  Yejin Choi,et al.  NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints , 2020, NAACL.

[26]  Minlie Huang,et al.  Dialogue Distillation: Open-domain Dialogue Augmentation Using Unpaired Data , 2020, EMNLP.

[27]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[28]  Hen-Hsen Huang,et al.  MPDD: A Multi-Party Dialogue Dataset for Analysis of Emotions and Interpersonal Relationships , 2020, LREC.

[29]  Mary Williamson,et al.  Recipes for Building an Open-Domain Chatbot , 2020, EACL.

[30]  Mary Williamson,et al.  Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills , 2020, ACL.

[31]  Haoyu Song,et al.  Generate, Delete and Rewrite: A Three-Stage Framework for Improving Persona Consistency of Dialogue Generation , 2020, ACL.

[32]  Qingyang Wu,et al.  TextGAIL: Generative Adversarial Imitation Learning for Text Generation , 2020, AAAI.

[33]  Dongyan Zhao,et al.  Low-Resource Knowledge-Grounded Dialogue Generation , 2020, ICLR.

[34]  Jeremy Blackburn,et al.  The Pushshift Reddit Dataset , 2020, ICWSM.

[35]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[36]  Haoyu Song,et al.  Generating Persona Consistent Dialogues by Exploiting Natural Language Inference , 2019, AAAI.

[37]  Zhiyuan Liu,et al.  Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs , 2019, ACL.

[38]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[39]  Steven Y. Feng,et al.  ALOHA: Artificial Learning of Human Attributes for Dialogue Agents , 2019, AAAI.

[40]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[41]  Dilek Z. Hakkani-Tür,et al.  Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations , 2019, INTERSPEECH.

[42]  Jason Weston,et al.  Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack , 2019, EMNLP.

[43]  Dongyan Zhao,et al.  Insufficient Data Can Also Rock! Learning to Converse Using Smaller Data with Augmentation , 2019, AAAI.

[44]  Zhou Yu,et al.  Domain Adaptive Dialog Generation via Meta Learning , 2019, ACL.

[45]  Boi Faltings,et al.  Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems , 2019, IJCAI.

[46]  Jason Weston,et al.  Dialogue Natural Language Inference , 2018, ACL.

[47]  Y-Lan Boureau,et al.  Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset , 2018, ACL.

[48]  Rada Mihalcea,et al.  MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations , 2018, ACL.

[49]  Jason Weston,et al.  Personalizing Dialogue Agents: I have a dog, do you have pets too? , 2018, ACL.

[50]  Xiaoyu Shen,et al.  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset , 2017, IJCNLP.

[51]  Jianfeng Gao,et al.  A Diversity-Promoting Objective Function for Neural Conversation Models , 2015, NAACL.

[52]  Joelle Pineau,et al.  The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems , 2015, SIGDIAL Conference.

[53]  Qinghua Zhu,et al.  Evaluation on crowdsourcing research: Current status and future direction , 2012, Information Systems Frontiers.

[54]  Elisa Bertino,et al.  Quality Control in Crowdsourcing Systems: Issues and Directions , 2013, IEEE Internet Computing.

[55]  Tomek Strzalkowski,et al.  MPC: A Multi-Party Chat Corpus for Modeling Social Phenomena in Discourse , 2010, LREC.

[56]  Karl G. D. Bailey,et al.  Do speakers and listeners observe the Gricean Maxim of Quantity , 2006 .

[57]  Sung Ju Hwang,et al.  Knowledge-Consistent Dialogue Generation with Knowledge Graphs , 2022 .

[58]  Daling Wang,et al.  Learning to Improve Persona Consistency in Multi-party Dialogue Generation via Text Knowledge Enhancement , 2022, COLING.

[59]  Hongbin Wang,et al.  Multi-Party Empathetic Dialogue Generation: A New Task for Dialog Systems , 2022, ACL.

[60]  Ning Ma,et al.  The CHiME corpus: a resource and a challenge for computational hearing in multisource environments , 2010, INTERSPEECH.