We present our Generative Enhanced Model (GEM) that we used to create samples awarded the first prize on the FEVER 2.0 Breakers Task. GEM is the extended language model developed upon GPT-2 architecture. The addition of novel target vocabulary input to the already existing context input enabled controlled text generation. The training procedure resulted in creating a model that inherited the knowledge of pretrained GPT-2, and therefore was ready to generate natural-like English sentences in the task domain with some additional control. As a result, GEM generated malicious claims that mixed facts from various articles, so it became difficult to classify their truthfulness.
[1]
Andreas Vlachos,et al.
Adversarial attacks against Fact Extraction and VERification
,
2019,
ArXiv.
[2]
Omer Levy,et al.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
,
2019,
ArXiv.
[3]
Ilya Sutskever,et al.
Language Models are Unsupervised Multitask Learners
,
2019
.
[4]
Vladimir I. Levenshtein,et al.
Binary codes capable of correcting deletions, insertions, and reversals
,
1965
.
[5]
Lukasz Kaiser,et al.
Attention is All you Need
,
2017,
NIPS.
[6]
Yiming Yang,et al.
XLNet: Generalized Autoregressive Pretraining for Language Understanding
,
2019,
NeurIPS.