Investigating Reasons for Disagreement in Natural Language Inference

Investigating Reasons for Disagreement in Natural Language Inference Nan-Jiang Jiang Department of Linguistics The Ohio State University, USA jiang.1879@osu.edu Marie-Catherine de Marneffe Department of Linguistics / FNRS The Ohio State University / UCLouvain USA / Belgium demarneffe.1@osu.edu Abstract We investigate how disagreement in natural language inference (NLI) annotation arises. We developed a taxonomy of disagreement sources with 10 categories spanning 3 high- level classes. We found that some disagree- ments are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts, leading to different interpretations of the label distribution. We explore two model- ing approaches for detecting items with po- tential disagreement: a 4-way classification with a ‘‘Complicated’’ label in addition to the three standard NLI labels, and a multi- label classification approach. We found that the multilabel classification is more expres- sive and gives better recall of the possible interpretations in the data. 1 Introduction Natural language inference (NLI) is the task of identifying whether a hypothesis sentence is true, false, or undetermined, given a premise. It is considered one of the most fundamental aspects of competent language understanding. In natural language processing, the NLI task is widely used to evaluate models’ semantic representations (i.a., Wang et al., 2019a), and to facilitate downstream tasks, for example, in natural language genera- tion (NLG). Large NLI datasets have been built by collect- ing inference judgments for premise-hypothesis pairs and aggregating the judgments by simple methods such as majority voting. However, it has been pointed out that NLI items do not all have a single ground truth and can exhibit system- atic disagreement (i.a., Pavlick and Kwiatkowski, 2019; Nie et al., 2020). This questions the as- sumption of having a single ground truth for each item and the validity of measuring models’ abil- ity to produce such ground truth. For instance, in (1) from the MNLI dataset (Williams et al., 2018), 3 out of 5 annotators labeled the item as ‘‘Entailment’’ (the hypothesis is inferred from the premise), 0 labeled it as ‘‘Neutral’’ (the hy- pothesis cannot be inferred from the premise), and 2 as ‘‘Contradiction’’ (the hypothesis con- tradicts the premise). (1) P: the only problem is it’s not large enough it only holds about i think they squeezed when Ryan struck out his five thousandth player they they squeezed about forty thousand people in there. H: It doesn’t hold many people. [E,N,C]: [3,0,2] People have indeed different judgments on which number is required to count as holding many people. The premise and hypothesis do not resolve explicitly what is being talked about, pos- sibly a stadium. Does 40,000 count as many for a stadium seating capacity? The premise states that it’s not large enough and uses the term squeez- ing, leading some annotators to see the hypothesis it doesn’t hold many people as being inferred from the premise. On the other hand, 40,000 peo- ple in a specific location is a large number, and some annotators therefore judge the hypothesis as contradictory to the premise. Such …

Content truncated to 500 words