NLP's Clever Hans Moment has Arrived

So far, so BERT, but here comes the twist: Instead of submitting yet another "we have SOTA, accept please" type paper, the authors were suspicious of this seemingly great success. Argument comprehension is a rather difficult task that requires world knowledge and commonsense reasoning (see figure below), and while no one doubts that BERT is one of the best language models created yet and that transfer learning is "NLP's Imagenet Moment", there is little evidence that language models are capable of such feats of highlevel natural language understanding.