FLUTE: Figurative Language Understanding through Textual Explanations

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139 - 7159 December 7-11, 2022 ©2022 Association for Computational Linguistics FLUTE: Figurative Language Understanding through Textual Explanations Tuhin Chakrabarty1 Arkady Saakyan1 Debanjan Ghosh2 Smaranda Muresan 1 1Department of Computer Science, Columbia University 2Educational Testing Service tuhin.chakr@cs.columbia.edu, a.saakyan@columbia.edu, dghosh@ets.org, smara@cs.columbia.edu Abstract Figurative language understanding has been recently framed as a recognizing textual en- tailment (RTE) task (a.k.a. natural language inference, or NLI). However, similar to classi- cal RTE/NLI datasets, the current benchmarks suffer from spurious correlations and annota- tion artifacts. To tackle this problem, work on NLI has built explanation-based datasets such as e-SNLI, allowing us to probe whether lan- guage models are right for the right reasons. Yet no such data exists for figurative language, making it harder to assess genuine understand- ing of such expressions. To address this issue, we release FLUTE, a dataset of 9,000 figura- tive NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms. We collect the data through a model-in-the-loop framework based on GPT- 3, crowd workers, and expert annotators. We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figura- tive language. The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to develop- ing models that understand figurative language through textual explanations. 1 Introduction Figurative language such as metaphors, similes or sarcasm plays an important role in enriching human communication, allowing us to express complex ideas and emotions in an implicit way (Roberts and Kreuz, 1994; Fussell and Moss, 1998). How- ever, understanding figurative language still re- mains a bottleneck for natural language processing (Shutova, 2011). Recently Jhamtani et al. (2021) show that when faced with dialog contexts con- sisting of figurative language,some models show very large drops in performance compared to con- texts without figurative language. Despite the fact that Transformer-based pre-trained language mod- els (LMs) get even larger (Brown et al., 2020; Raf- fel et al., 2020), they are still unable to comprehend the physical world, cultural knowledge, or social context in which figurative language is embedded (Bisk et al., 2020). In recent years, there have been several bench- marks dedicated to figurative language understand- ing, which generally frame “understanding” as a recognizing textual entailment (a.k.a natural lan- guage inference (NLI)) task — deciding whether one sentence (premise) entails/contradicts another (hypothesis) (Chakrabarty et al., 2021; Stowe et al., 2022; Srivastava et al., 2022). However, similar to general NLI datasets, these benchmarks suffer from spurious correlations and annotation artifacts (Mc- Coy et al., 2019; Poliak et al., 2018b). These can allow large language models (LLMs) to achieve near human-level performance on in-domain test sets, yet turn brittle when evaluated against out-of- domain or adversarial examples (Glockner et al., 2018; Ribeiro et al., 2016, 2020). To tackle these problems, research in NLI has argued that it is …

Content truncated to 500 words