New Protocols and Negative Results for Textual Entailment Data Collection

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8203–8214, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8203 New Protocols and Negative Results for Textual Entailment Data Collection Samuel R. Bowman∗ Center for Data Science, Department of Linguistics, and Department of Computer Science New York University bowman@nyu.edu Jennimaria Palomaki, Livio Baldini Soares, Emily Pitler Google Research {jpalomaki,liviobs,epitler}@google.com Abstract Natural language inference (NLI) data has proven useful in benchmarking and, especially, as pretraining data for tasks requiring language understanding. However, the crowdsourcing protocol that was used to collect this data has known issues and was not explicitly optimized for either of these purposes, so it is likely far from ideal. We propose four alternative proto- cols, each aimed at improving either the ease with which annotators can produce sound train- ing examples or the quality and diversity of those examples. Using these alternatives and a fifth baseline protocol, we collect and compare five new 8.5k-example training sets. In evalua- tions focused on transfer learning applications, our results are solidly negative, with models trained on our baseline dataset yielding good transfer performance to downstream tasks, but none of our four new methods (nor the recent ANLI) showing any improvements over that baseline. In a small silver lining, we observe that all four new protocols, especially those where annotators edit pre-filled text boxes, re- duce previously observed issues with annota- tion artifacts. 1 Introduction The task of natural language inference (NLI; also known as textual entailment) has been widely used as an evaluation task when developing new meth- ods for language understanding tasks, and it has recently become clear that high-quality NLI data can be useful in transfer learning as well, driv- ing much of the recent use of the task: Several recent papers have shown that training large neu- ral network models on natural language inference data, then fine-tuning them for other language un- derstanding tasks often yields substantially better results on those target tasks (Conneau et al., 2017; Subramanian et al., 2018). This result holds even ∗Work done while visiting Google. Base Premise: –––––––––––––, –––––. entailment: contradiction: neutral: | Paragraph Premise: –––––––––––––, –––––. –––––––––– –– (––). –– –––––––, ––––, –––––. –––––––––– –––, –––– –––––. ––– –––––. –––––––––. –– –––––––, –––––––, –––––. entailment: contradiction: neutral: | EditPremise Premise: –––––––––––––, –––––. entailment: contradiction: neutral: –––––––––––––, –––––.| –––––––––––––, –––––. –––––––––––––, –––––. EditOther Premise: –––––––––––––, –––––. entailment: contradiction: neutral: –––––––––––, ––, ––––––––.| –––––––––––, ––, ––––––––. –––––––––––, ––, ––––––––. Contrast Main Premise: –––––––––––––, –––––. Contrasting Premise: –––––––––, ––, ––––––––. entailment: contradiction: | Figure 1: The annotation interfaces we evaluate. when starting from large models like BERT (De- vlin et al., 2019) that have already been pretrained extensively on unlabeled data (Phang et al., 2018; Clark et al., 2019; Liu et al., 2019b; Wang et al., 2019b). The largest general-purpose corpus for NLI, and the one that has proven most successful in this setting, is the Multi-Genre NLI Corpus (MNLI Williams et al., 2018). MNLI was designed infor- mally for use in …

Content truncated to 500 words