Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 672–686 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 672 Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options Clara Vania Ruijie Chen Samuel R. Bowman New York University {c.vania, rc3959, bowman}@nyu.edu Abstract Large-scale natural language inference (NLI) datasets such as SNLI or MNLI have been created by asking crowdworkers to read a premise and write three new hypotheses, one for each possible semantic relationships (en- tailment, contradiction, and neutral). While this protocol has been used to create useful benchmark data, it remains unclear whether the writing-based annotation protocol is opti- mal for any purpose, since it has not been eval- uated directly. Furthermore, there is ample evidence that crowdworker writing can intro- duce artifacts in the data. We investigate two alternative protocols which automatically cre- ate candidate (premise, hypothesis) pairs for annotators to label. Using these protocols and a writing-based baseline, we collect several new English NLI datasets of over 3k exam- ples each, each using a fixed amount of an- notator time, but a varying number of exam- ples to fit that time budget. Our experiments on NLI and transfer learning show negative re- sults: None of the alternative protocols outper- forms the baseline in evaluations of generaliza- tion within NLI or on transfer to outside target tasks. We conclude that crowdworker writing still the best known option for entailment data, highlighting the need for further data collec- tion work to focus on improving writing-based annotation processes. 1 Introduction Research on natural language understanding has benefited greatly from the availability of large- scale, annotated data, especially for tasks like read- ing comprehension and natural language inference, which lend themselves to non-expert crowdsourc- ing. These datasets are useful in three settings: evaluation (Williams et al., 2018; Rajpurkar et al., 2018; Zellers et al., 2019); pretraining (Phang et al., 2018; Conneau et al., 2018; Pruksachatkun et al., 2020); and as training data for downstream tasks (Trivedi et al., 2019; Portelli et al., 2020). Natural language inference (NLI), also known as recognizing textual entailment (RTE; Dagan et al., 2005) is the problem of determining whether or not a hypothesis semantically entails a premise. The two largest NLI corpora, SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) are created by asking crowdworkers to write three labeled hy- pothesis sentences given a premise sentence taken from a preexisting text corpus. While these datasets have been widely used as benchmarks for NLU, there have been no studies evaluating writing-based annotation for collecting NLI data. Moreover, there is growing evidence that human writing can intro- duce annotation artifacts, which enable models to perform moderately well just by learning spurious statistical patterns in the data (Gururangan et al., 2018; Tsuchiya, 2018; Poliak et al., 2018a). This paper explores the possibility of collecting high-quality NLI data without asking crowdwork- ers to write hypotheses. We …

Content truncated to 500 words