Does the “Most Sinfully Decadent Cake Ever” Taste Good? Answering Yes/No Questions from Figurative Contexts

Proceedings of Recent Advances in Natural Language Processing, pages 926–936 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_100 926 Does the “most sinfully decadent cake ever” taste good? Answering Yes/No Questions from Figurative Contexts Geetanjali Rakshit and Jeffrey Flanigan Computer Science and Engineering Department UC Santa Cruz {grakshit,jmflanig}@ucsc.edu Abstract Figurative language is commonplace in natural language, and while making communication memorable and creative, can be difficult to un- derstand. In this work, we investigate the ro- bustness of Question Answering (QA) models on figurative text. Yes/no questions, in partic- ular, are a useful probe of figurative language understanding capabilities of large language models. We propose FigurativeQA, a set of 1000 yes/no questions with figurative and non- figurative contexts, extracted from the domains of restaurant and product reviews. We show that state-of-the-art BERT-based QA models exhibit an average performance drop of up to 15% points when answering questions from fig- urative contexts, as compared to non-figurative ones. While models like GPT-3 and ChatGPT are better at handling figurative texts, we show that further performance gains can be achieved by automatically simplifying the figurative con- texts into their non-figurative (literal) counter- parts. We find that the best overall model is ChatGPT with chain-of-thought prompting to generate non-figurative contexts. Our work pro- vides a promising direction for building more robust QA models with figurative language un- derstanding capabilities. 1 Introduction “Questions are never indiscreet. Answers sometimes are.” - Oscar Wilde One of the many interesting phenomena occur- ring in natural language is the presence of figurative language, which, while making communication creative and memorable (Danescu-Niculescu-Mizil et al., 2012), may sometimes also prove difficult to understand (Zayed et al., 2020). This includes (but is not limited to) linguistic constructs such as idioms, similes, metaphors, rhetorical questions, hyperbole, personification, sarcasm, and irony. It The cake was described as the most sinfully decadent ever . Question: Did the cake taste good? Answer: Yes Figure 1: To answer the question “Did the cake taste good?” based on the context, a Question Answering (QA) model needs to be able to correctly infer the mean- ing of the figurative text “the most sinfully decadent ever” may be particularly difficult for non-native speak- ers to interpret figurative expressions, and phenom- ena like sarcasm are often missed altogether (Joshi et al., 2016). Given that figurativeness is com- monplace in everyday communication (Lakoff and Johnson, 2008), progress in the field of Natural Language Understanding (NLU) would be incom- plete without figurativeness understanding. Conse- quently, figurative text has been studied in various downstream NLP tasks such as machine translation (Dankers et al., 2022), textual entailment (Agerri, 2008), (Chakrabarty et al., 2021), (Liu et al., 2022) and dialog models (Jhamtani et al., 2021), inter-alia. However, to the best of our knowledge, there has not been a systematic study of figurative language understanding capabilities of question answering models. We focus on yes/no questions for our question answering (QA) task. Yes/no questions are a good test of figurative language understanding because correctly answering them requires the reader to correctly …

Content truncated to 500 words