ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3989 - 4004 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding Sayan Ghosh Shashank Srivastava UNC Chapel Hill {sayghosh,ssrivastava}@cs.unc.edu Abstract While large language models have shown ex- citing progress on several NLP benchmarks, evaluating their ability for complex analogical reasoning remains under-explored. Here, we introduce a high-quality crowdsourced dataset of narratives for employing proverbs in con- text as a benchmark for abstract language un- derstanding. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and contains minimal lexical overlaps between narratives and proverbs, en- suring that models need to go beyond surface- level reasoning to succeed. We explore three tasks: (1) proverb recommendation and align- ment prediction, (2) narrative generation for a given proverb and topic, and (3) identify- ing narratives with similar motifs. Our experi- ments show that neural language models strug- gle on these tasks compared to humans, and these tasks pose multiple learning challenges. 1 Introduction Large language models (LLMs) (Devlin et al., 2019; Liu et al., 2019a; Raffel et al., 2020; Lewis et al., 2020; Reimers and Gurevych, 2019; Sanh et al., 2019; Lan et al., 2020) have led to a paradigm shift in NLP, and have shown exciting progress on benchmarks such as GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a). In particu- lar, these include tasks such as reading comprehen- sion, natural language inference, and coreference resolution. Many of these tasks rely on semantic and syntactic reasoning, which has been mastered by these LLMs. For example, apart from improv- ing on distributional semantics through contextual- ized embeddings (Ethayarajh, 2019), recent work has shown evidence that these models implicitly learn emergent concepts such as subject-verb agree- ment (Jawahar et al., 2019), semantic roles (Tenney et al., 2019) and dependency structures (Hewitt and Manning, 2019). Prevention is better than cure Instead of working on his English assignments weekly, he put them off until the last week before they were all due. He was able to finish them all over the final few days, but it took a lot of energy drinks and misery. There was once a disease that spread like wildfire. It killed people by the thousands. Doctors said wash your hands regularly and you'll be ok. Eventually an inoculation shot was created, and it worked, except on the people who refused to wash their hands and died waiting for their turn to get a shot. NARRATIVE (N2) NARRATIVE (N1) PROVERB (P) Keywords (K1) : { assignments, working, misery } Keywords (K2) : { disease, washing, hands, inoculation } Figure 1: We introduce ePiC, a crowdsourced dataset of narratives for employing proverbs in context. Our dataset contains narratives (N1 and N2) paired against proverbs (P) along with a fine-grained annotation of aligned spans between the narratives and proverbs. Aligned spans are shown with matching colors and in- dicate correspondences in …

Content truncated to 500 words