Capture Human Disagreement Distributions by Calibrated Networks for Natural Language Inference

Findings of the Association for Computational Linguistics: ACL 2022, pages 1524 - 1535 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Capture Human Disagreement Distributions by Calibrated Networks for Natural Language Inference Yuxia Wang1∗, Minghan Wang 2, Yimeng Chen 2, Shimin Tao 2, Jiaxin Guo 2, Chang Su 2, Min Zhang2, Hao Yang 2 1 The University of Melbourne, Victoria, Australia 2 Huawei Translation Services Center, Beijing, China yuxiaw@student.unimelb.edu.au {wangyuxia5,chenyimeng,taoshimin, guojiaxin1,suchang8,zhangmin186,yanghao30}@huawei.com Abstract Natural Language Inference (NLI) datasets con- tain examples with highly ambiguous labels due to its subjectivity. Several recent efforts have been made to acknowledge and embrace the existence of ambiguity, and explore how to capture the human disagreement distribu- tion. In contrast with directly learning from gold ambiguity labels, relying on special re- source, we argue that the model has naturally captured the human ambiguity distribution as long as it’s calibrated, i.e. the predictive prob- ability can reflect the true correctness likeli- hood. Our experiments show that when model is well-calibrated, either by label smoothing or temperature scaling, it can obtain competitive performance as prior work, on both divergence scores between predictive probability and the true human opinion distribution, and the accu- racy. This reveals the overhead of collecting gold ambiguity labels can be cut, by broadly solving how to calibrate the NLI network. 1 Introduction Ambiguity is intrinsic to natural language. Previ- ously, it’s common to disregard it as noise or as a sign of poor-quality data, because we implicitly make the assumption that there is only one correct label given an example, indicating the unique class it belongs to. However, it is against the subjectiv- ity of many natural language understanding (NLU) tasks, such as natural language inference (NLI) and semantic textual similarity (STS), as their anno- tations are heavily based on personal experience and opinions. More recent research has gravitated towards the necessity to acknowledge and embrace the existence of ambiguity in NLI. Pavlick and Kwiatkowski (2019) shows that hu- man disagreements, very often, are not dismissible as annotation noise, but rather persist as collecting more ratings and varying the amount of context ∗* Work carried out as an intern at Huawei 2012 Lab. Premise Look, there’s a legend here. Hypothesis See, there is a well known hero here. Label C: 1, E: 57, N: 42 Source Chaos-MultiNLI Premise A group of onlookers glance at a person doing a strange trick on her head. Hypothesis A boy does a card trick. Label C: 56, E: 1, N: 43 Source Chaos-SNLI Table 1: Ambiguous examples from ChaosNLI. Label: the first element is the class, the second is the number of annotators among 100 choosing this class. C E N refers to classes of Contradiction, Entailment and Neutral. provided to raters. Humans judgments cannot be adequately summarized by a single aggregate label or value, 1 but a distribution. ChaosNLI (Nie et al., 2020) provides an empirical distribution by collect- ing 100 annotations for each instance, to simulate true soft label distribution, which is always used as the …

Content truncated to 500 words