Uncertain Natural Language Inference

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8772 Uncertain Natural Language Inference Tongfei Chen1∗ Zhengping Jiang2∗† Adam Poliak1 Keisuke Sakaguchi3† Benjamin Van Durme1 1 Johns Hopkins University 2 Columbia University 3 Allen Institute for AI {tongfei,azpoliak,vandurme}@jhu.edu zj2265@columbia.edu, keisukes@allenai.org Abstract We introduce Uncertain Natural Language In- ference (UNLI), a refinement of Natural Lan- guage Inference (NLI) that shifts away from categorical labels, targeting instead the di- rect prediction of subjective probability assess- ments. We demonstrate the feasibility of col- lecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilis- tic scale, where items even with the same cate- gorical label differ in how likely people judge them to be true given a premise. We describe a direct scalar regression modeling approach, and find that existing categorically labeled NLI data can be used in pre-training. Our best models approach human performance, demon- strating models may be capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks. 1 Introduction Variants of entailment tasks have been used for decades in benchmarking systems for natural lan- guage understanding. Recognizing Textual Entail- ment (RTE) or Natural Language Inference (NLI) is traditionally a categorical classification problem: predict which of a set of discrete labels apply to an inference pair, consisting of a premise (푝) and hypothesis (ℎ). The FraCaS consortium offered the task as an evaluation mechanism, along with a small challenge set (Cooper et al., 1996), which was followed by the RTE challenges (Dagan et al., 2005). Despite differences between these and re- cent NLI datasets (Marelli et al., 2014; Lai et al., 2017; Williams et al., 2018; Khot et al., 2018, i.a.), NLI hsa remained a categorical prediction problem. However, entailment inference is uncertain and has a probabilistic nature (Glickman et al., 2005). Maintaining NLI as a categorical classification ∗ Equal contribution. † Work performed while at Johns Hopkins University. Premise � Hypothesis NLI UNLI A man in a white shirt taking a picture � A man takes a picture ENT 100% A boy hits a ball, with a bat � The kid is playing in a baseball game ENT 78% A wrestler in red cries, one in blue celebrates � The wrestler in blue is undefeated CON 50% Man laying on a platform outside on rocks � Man takes a nap on his couch CON 0% Table 1: Probability assessments on NLI pairs. The NLI and UNLI columns respectively indicate the cate- gorical label (from SNLI) and the subjective probabil- ity for the corresponding pair. problem is not ideal since coarse categorical la- bels mask the uncertain and probabilistic nature of entailment inference. NLI pairs may share a coarse label, but the probabilities that the hypotheses are entailed by their corresponding premises may vary greatly (see Table 1). Hence, not all contradictions are equally contradictory and not all entailments are equally entailed. We propose Uncertain Natural Language …

Content truncated to 500 words