Advancements in Scientific Controllable Text Generation Methods

The previous work on controllable text generation is organized using a new schema we provide in this study. Seven components make up the schema, and each one is crucial to the creation process. To accomplish controlled generation for scientific literature, we describe the various modulation strategies utilised to modulate each of the seven components. We also offer a theoretical study and qualitative examination of these methods. This insight makes possible new architectures based on combinations of these components. Future research will compare these methods empirically to learn more about their strengths and utility.

[1]  Christina Y. Yu,et al.  Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language model , 2023, ArXiv.

[2]  Jieyi Long.  Large Language Model Guided Tree-of-Thought , 2023, ArXiv.

[3]  Zhou Yu,et al.  Mixture of Soft Prompts for Controllable Data Generation , 2023, ArXiv.

[4]  Naman Goyal,et al.  LLaMA: Open and Efficient Foundation Language Models , 2023, ArXiv.

[5]  Michel Galley,et al.  Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback , 2023, ArXiv.

[6]  Tong Zhang,et al.  Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data , 2023, EMNLP.

[7]  Michel Galley,et al.  Guiding Large Language Models via Directional Stimulus Prompting , 2023, ArXiv.

[8]  Noah A. Smith,et al.  Self-Instruct: Aligning Language Models with Self-Generated Instructions , 2022, ACL.

[9]  Guillem Cucurull,et al.  Galactica: A Large Language Model for Science , 2022, ArXiv.

[10]  Richard H. R. Hahnloser,et al.  Controllable Citation Text Generation , 2022, ArXiv.

[11]  Alexander J. Smola,et al.  Automatic Chain of Thought Prompting in Large Language Models , 2022, ICLR.

[12]  D. Schuurmans,et al.  Least-to-Most Prompting Enables Complex Reasoning in Large Language Models , 2022, ICLR.

[13]  S. Yuan,et al.  Intent-Controllable Citation Text Generation , 2022, Mathematics.

[14]  Angeliki Lazaridou,et al.  Internet-augmented language models through few-shot prompting for open-domain question answering , 2022, ArXiv.

[15]  Dale Schuurmans,et al.  Chain of Thought Prompting Elicits Reasoning in Large Language Models , 2022, NeurIPS.

[16]  Alexander Te-Wei Shieh,et al.  Towards Generating Citation Sentences for Multiple References with Intent Control , 2021, ArXiv.

[17]  Brian Lester,et al.  The Power of Scale for Parameter-Efficient Prompt Tuning , 2021, EMNLP.

[18]  Zhengxiao Du,et al.  GPT Understands, Too , 2021, AI Open.

[19]  Ryan J. Lowe,et al.  Learning to summarize from human feedback , 2020, NeurIPS 2020.

[20]  Àlex Bravo,et al.  Automatic related work section generation: experiments in scientific document abstracting , 2020, Scientometrics.

[21]  Xiaojun Wan,et al.  Automatic Generation of Citation Texts in Scholarly Papers: A Pilot Study , 2020, ACL.

[22]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[23]  Alan W Black,et al.  Exploring Controllable Text Generation Techniques , 2020, COLING.

[24]  Pengfei Liu,et al.  Extractive Summarization as Text Matching , 2020, ACL.

[25]  Arman Cohan,et al.  Longformer: The Long-Document Transformer , 2020, ArXiv.

[26]  Noah A. Smith,et al.  Explaining Relationships Between Scientific Documents , 2020, ACL.

[27]  Omer Levy,et al.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension , 2019, ACL.

[28]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[29]  Alan W Black,et al.  Augmenting Non-Collaborative Dialog Systems with Explicit Semantic and Strategic Dialog History , 2019, ICLR.

[30]  J. Yosinski,et al.  Plug and Play Language Models: A Simple Approach to Controlled Text Generation , 2019, ICLR.

[31]  Jason Weston,et al.  I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents , 2019, ArXiv.

[32]  Tom B. Brown,et al.  Fine-Tuning Language Models from Human Preferences , 2019, ArXiv.

[33]  Lav R. Varshney,et al.  CTRL: A Conditional Transformer Language Model for Controllable Generation , 2019, ArXiv.

[34]  Akhilesh Sudhakar,et al.  “Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer , 2019, EMNLP.

[35]  Sameer Singh,et al.  Universal Adversarial Triggers for Attacking and Analyzing NLP , 2019, EMNLP.

[36]  Iryna Gurevych,et al.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks , 2019, EMNLP.

[37]  Ido Dagan,et al.  Better Rewards Yield Better Summaries: Learning to Summarise Without References , 2019, EMNLP.

[38]  Jason Weston,et al.  Neural Text Generation with Unlikelihood Training , 2019, ICLR.

[39]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[40]  Xiaojun Wan,et al.  Controllable Unsupervised Text Attribute Transfer via Editing Entangled Latent Representation , 2019, NeurIPS.

[41]  Eric P. Xing,et al.  Target-Guided Open-Domain Conversation , 2019, ACL.

[42]  Dilek Z. Hakkani-Tür,et al.  Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators , 2019, INLG.

[43]  Yejin Choi,et al.  The Curious Case of Neural Text Degeneration , 2019, ICLR.

[44]  Waleed Ammar,et al.  Structural Scaffolds for Citation Intent Classification in Scientific Publications , 2019, NAACL.

[45]  Eric Horvitz,et al.  Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting , 2019, DGS@ICLR.

[46]  Jason Weston,et al.  Learning to Speak and Act in a Fantasy Text Adventure Game , 2019, EMNLP.

[47]  Kyunghyun Cho,et al.  Non-Monotonic Sequential Text Generation , 2019, ICML.

[48]  Jaewoo Kang,et al.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining , 2019, Bioinform..

[49]  Samy Bengio,et al.  Content preserving text generation with attribute controls , 2018, NeurIPS.

[50]  J. Weston,et al.  Wizard of Wikipedia: Knowledge-Powered Conversational agents , 2018, ICLR.

[51]  Alan W. Black,et al.  A Dataset for Document Grounded Conversations , 2018, EMNLP.

[52]  Anna Rumshisky,et al.  Adversarial Decomposition of Text Representation , 2018, NAACL.

[53]  Mohit Bansal,et al.  Polite Dialogue Generation Without Parallel Data , 2018, TACL.

[54]  Yann Dauphin,et al.  Hierarchical Neural Story Generation , 2018, ACL.

[55]  Yejin Choi,et al.  Learning to Write with Cooperative Discriminators , 2018, ACL.

[56]  Yuxiang Wu,et al.  Learning to Extract Coherent Summary via Deep Reinforcement Learning , 2018, AAAI.

[57]  Joel R. Tetreault,et al.  Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer , 2018, NAACL.

[58]  Jason Weston,et al.  Personalizing Dialogue Agents: I have a dog, do you have pets too? , 2018, ACL.

[59]  Dongyan Zhao,et al.  Style Transfer in Text: Exploration and Evaluation , 2017, AAAI.

[60]  Xing Shi,et al.  Hafez: an Interactive Poetry Generation System , 2017, ACL.

[61]  Hal Daumé,et al.  Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback , 2017, EMNLP.

[62]  Shane Legg,et al.  Deep Reinforcement Learning from Human Preferences , 2017, NIPS.

[63]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[64]  Yang Liu,et al.  Learning Structured Text Representations , 2017, TACL.

[65]  Richard Socher,et al.  A Deep Reinforced Model for Abstractive Summarization , 2017, ICLR.

[66]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[67]  Victor O. K. Li,et al.  Trainable Greedy Decoding for Neural Machine Translation , 2017, EMNLP.

[68]  Ming-Wei Chang,et al.  A Knowledge-Grounded Neural Conversation Model , 2017, AAAI.

[69]  Daniel Jurafsky,et al.  Learning to Decode for Future Success , 2017, ArXiv.

[70]  Quoc V. Le,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[71]  Jianfeng Gao,et al.  A Persona-Based Neural Conversation Model , 2016, ACL.

[72]  S. Chopra,et al.  Sequence Level Training with Recurrent Neural Networks , 2015, ICLR.

[73]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[74]  David Vandyke,et al.  Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems , 2015, EMNLP.

[75]  Yoshua Bengio,et al.  Attention-Based Models for Speech Recognition , 2015, NIPS.

[76]  Xiaojun Wan,et al.  Automatic Generation of Related Work Sections in Scientific Papers: An Optimization Approach , 2014, EMNLP.

[77]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[78]  Min-Yen Kan,et al.  Towards Automated Related Work Summarization , 2010, COLING.

[79]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[80]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[81]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[82]  Percy Liang,et al.  Prefix-Tuning: Optimizing Continuous Prompts for Generation , 2021, ACL.

[83]  Dongyan Zhao,et al.  Capturing Relations between Scientific Papers: An Abstractive Model for Related Work Section Generation , 2021, ACL.

[84]  Jana Diesner,et al.  BACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation , 2021, ACL.

[85]  Yulia Tsvetkov,et al.  StructSum: Summarization via Structured Representations , 2021, EACL.

[86]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[87]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[88]  Nanyun Peng,et al.  Towards Controllable Story Generation , 2018 .

[89]  Xinyan Xiao,et al.  Improving Neural Abstractive Document Summarization with Explicit Information Selection Modeling , 2018, EMNLP.

[90]  Christopher S. G. Khoo,et al.  Literature Review Writing: A Study of Information Selection from Cited Papers , 2011 .