Identifying inherent disagreement in natural language inference

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4908–4915 June 6–11, 2021. ©2021 Association for Computational Linguistics 4908 Identifying inherent disagreement in natural language inference Xinliang Frederick Zhang Department of Computer Science and Engineering The Ohio State University zhang.9975@osu.edu Marie-Catherine de Marneffe Department of Linguistics The Ohio State University demarneffe.1@osu.edu Abstract Natural language inference (NLI) is the task of determining whether a piece of text is en- tailed, contradicted by or unrelated to another piece of text. In this paper, we investigate how to tease systematic inferences (i.e., items for which people agree on the NLI label) apart from disagreement items (i.e., items which lead to different annotations), which most prior work has overlooked. To distinguish sys- tematic inferences from disagreement items, we propose Artificial Annotators (AAs) to sim- ulate the uncertainty in the annotation process by capturing the modes in annotations. Re- sults on the CommitmentBank, a corpus of nat- urally occurring discourses in English, confirm that our approach performs statistically signif- icantly better than all baselines. We further show that AAs learn linguistic patterns and context-dependent reasoning. 1 Introduction Learning to effectively understand unstructured text is integral to Natural Language Understand- ing (NLU), covering a wide range of tasks such as question answering, semantic textual similarity and sentiment analysis. Natural language infer- ence (NLI), an increasingly important benchmark task for NLU research, is the task of determining whether a piece of text is entailed, contradicted by or unrelated to another piece of text (i.a., Dagan et al., 2005; MacCartney and Manning, 2009). Pavlick and Kwiatkowski (2019) observed inher- ent disagreements among annotators in several NLI datasets, which cannot be smoothed out by hiring more people. They pointed out that to achieve ro- bust NLU, we need to be able to tease apart system- atic inferences (i.e., items for which most people agree on the annotations) from items inherently leading to disagreement. The last example in Ta- ble 1, from the CommitmentBank (de Marneffe et al., 2019), is a typical disagreement item: some annotators consider it to be an entailment (3 or 2), 1 Premise: Some of them, like for instance the farm in Connecticut, are quite small. If I like a place I buy it. I guess you could say it’s a hobby. Hypothesis: buying places is a hobby. Entailment (Entailment) [3, 3, 2, 2, 2, 2, 1, 1] 2 Premise: “I hope you are settling down and the cat is well.” This was a lie. She did not hope the cat was well. Hypothesis: the cat was well. Neutral (Neutral) [0, 0, 0, 0, 0, 0, 0, 0, -3] 3 Premise: “All right, so it wasn’t the bottle by the bed. What was it, then?” Cobalt shook his head which might have meant he didn’t know or might have been admonishment for Oliver who was still holding the bottle of wine. Hypothesis: Cobalt didn’t know. Neutral (Disagreement) [1, 0, 0, 0, 0, 0, 0, -2] …

Content truncated to 500 words