Adversarial NLI: A New Benchmark for Natural Language Understanding

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4885 Adversarial NLI: A New Benchmark for Natural Language Understanding Yixin Nie∗, Adina Williams†, Emily Dinan†, Mohit Bansal∗, Jason Weston†, Douwe Kiela† ∗UNC Chapel Hill †Facebook AI Research Abstract We introduce a new large-scale NLI bench- mark dataset, collected via an iterative, ad- versarial human-and-model-in-the-loop proce- dure. We show that training models on this new dataset leads to state-of-the-art perfor- mance on a variety of popular NLI bench- marks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the- art models, and shows that non-expert annota- tors are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. 1 Introduction Progress in AI has been driven by, among other things, the development of challenging large-scale benchmarks like ImageNet (Russakovsky et al., 2015) in computer vision, and SNLI (Bowman et al., 2015), SQuAD (Rajpurkar et al., 2016), and others in natural language processing (NLP). Re- cently, for natural language understanding (NLU) in particular, the focus has shifted to combined benchmarks like SentEval (Conneau and Kiela, 2018) and GLUE (Wang et al., 2018), which track model performance on multiple tasks and provide a unified platform for analysis. With the rapid pace of advancement in AI, how- ever, NLU benchmarks struggle to keep up with model improvement. Whereas it took around 15 years to achieve “near-human performance” on MNIST (LeCun et al., 1998; Cires¸an et al., 2012; Wan et al., 2013) and approximately 7 years to surpass humans on ImageNet (Deng et al., 2009; Russakovsky et al., 2015; He et al., 2016), the GLUE benchmark did not last as long as we would have hoped after the advent of BERT (Devlin et al., 2018), and rapidly had to be extended into Super- GLUE (Wang et al., 2019). This raises an important question: Can we collect a large benchmark dataset that can last longer? The speed with which benchmarks become ob- solete raises another important question: are cur- rent NLU models genuinely as good as their high performance on benchmarks suggests? A grow- ing body of evidence shows that state-of-the-art models learn to exploit spurious statistical patterns in datasets (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Glockner et al., 2018; Geva et al., 2019; McCoy et al., 2019), instead of learn- ing meaning in the flexible and generalizable way that humans do. Given this, human annotators—be they seasoned NLP researchers or non-experts— might easily be able to construct examples that expose model brittleness. We propose an iterative, adversarial human-and- model-in-the-loop solution for NLU dataset collec- tion that addresses both benchmark longevity and robustness issues. In the first stage, human anno- tators devise examples that our current best mod- els cannot determine the correct label …

Content truncated to 500 words