HellaSwag: Can a Machine Really Finish Your Sentence?

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4791 HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers♠ Ari Holtzman♠ Yonatan Bisk♠ Ali Farhadi♠~ Yejin Choi♠~ ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ~Allen Institute for Artificial Intelligence https://rowanzellers.com/hellaswag Abstract Recent work by Zellers et al. (2018) intro- duced a new task of commonsense natural lan- guage inference: given an event description such as “A woman sits at a piano,” a machine must select the most likely followup: “She sets her fingers on the keys.” With the intro- duction of BERT (Devlin et al., 2018), near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense in- ference still proves difficult for even state- of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its ques- tions are trivial for humans (°95% accuracy), state-of-the-art models struggle (†48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key in- sight is to scale up the length and complex- ity of the dataset examples towards a critical ‘Goldilocks’ zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its result- ing difficulty, sheds light on the inner work- ings of deep pretrained models. More broadly, it suggests a new path forward for NLP re- search, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges. 1 Introduction Imagine a woman chasing a dog around outside, trying to give it a bath. What might happen next? Humans can read a narrative like this, shown in Figure 1, and connect it to a rich model of the world: the dog is currently dry and not soapy, and it actively doesn’t want to be bathed. Thus, one A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. She… A. rinses the bucket off with soap and blow dry the dog’s head. B. uses a hose to keep it from getting soapy. C. gets the dog wet, then it runs away again. D. gets into a bath tub with the dog. Come to a complete halt at a stop sign or red light. At a stop sign, come to a complete halt for about 2 seconds or until vehicles that arrived before you clear the intersection. If you're stopped at a red light, proceed when the light has turned green. … A. Stop for no more than two seconds, or until the light turns yellow. A red light in front of you indicates that you should stop. B. After you come to a complete stop, turn off your turn …

Content truncated to 500 words