IMPLI: Investigating NLI Models’ Performance on Figurative Language

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5375 - 5388 May 22-27, 2022 c⃝2022 Association for Computational Linguistics IMPLI: Investigating NLI Models’ Performance on Figurative Language Kevin Stowe♦ Prasetya Ajie Utama†∗ Iryna Gurevych♦ ♦ Ubiquitous Knowledge Processing Lab (UKP) Department of Computer Science Technical University of Darmstadt † Bloomberg, London, United Kingdom www.ukp.tu-darmstadt.de Abstract Natural language inference (NLI) has been widely used as a task to train and evaluate mod- els for language understanding. However, the ability of NLI models to perform inferences requiring understanding of figurative language such as idioms and metaphors remains under- studied. We introduce the IMPLI (Idiomatic and Metaphoric Paired Language Inference) dataset, an English dataset consisting of paired sentences spanning idioms and metaphors. We develop novel methods to generate 24k semi- automatic pairs as well as manually creating 1.8k gold pairs. We use IMPLI to evaluate NLI models based on RoBERTa fine-tuned on the widely used MNLI dataset. We then show that while they can reliably detect entailment rela- tionship between figurative phrases with their literal counterparts, they perform poorly on similarly structured examples where pairs are designed to be non-entailing. This suggests the limits of current NLI models with regard to understanding figurative language and this dataset serves as a benchmark for future im- provements in this direction.1 1 Introduction Understanding figurative language (i.e., that in which the intended meaning of the utterance dif- fers from the literal compositional meaning) is a particularly difficult area in NLP (Shutova, 2011; Veale et al., 2016), but is essential for proper natu- ral language understanding. We consider here two types of figurative language: idioms and metaphors. Idioms can be viewed as non-compositional multi- word expressions (Jochim et al., 2018), and have been historically difficult for NLP systems. For in- stance, sentiment systems struggle with multiword expressions in which individual words do not di- rectly contribute to the sentiment (Sag et al., 2002). ∗ The work was done while the second author was still affiliated with the UKP Lab at TU Darmstadt. 1Dataset and all related resources are publicly available at https://github.com/UKPLab/acl2022-impli. Idioms Jamie was pissed off this afternoon. → Jamie was irritated this afternoon There’s a marina down in the docks. ↛ There’s a marina down under scrutiny. Metaphors The hearts of men were softened. → The men were made kindler and gentler. The gun kicked into my shoulder. ↛ The mule kicked into my shoulder. Table 1: Examples of entailment (→) and non- entailment pairs (↛) from the IMPLI dataset. Metaphors involve linking conceptual properties of two or more domains, and are known to be per- vasive in everyday language (Lakoff and Johnson, 1980; Stefanowitsch and Gries, 2008; Steen et al., 2010). Recent work has shown that these types of figurative language are impactful across a broad array of NLP tasks (see §2.1). Large-scale pre-training and transformer-based architectures have yielded increasingly powerful language models (Vaswani et al., 2017; Devlin et al., 2019; Liu et al., 2019). …

Content truncated to 500 words