Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions

Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions Ziheng Zeng and Suma Bhat Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign Champaign, IL USA {zzeng13, spbhat2}@illinois.edu Abstract Idiomatic expressions (IEs), characterized by their non-compositionality, are an impor- tant part of natural language. They have been a classical challenge to NLP, including pre-trained language models that drive today’s state-of-the-art. Prior work has identified defi- ciencies in their contextualized representation stemming from the underlying compositional paradigm of representation. In this work, we take a first-principles approach to build idiomaticity into BART using an adapter as a lightweight non-compositional language expert trained on idiomatic sentences. The im- proved capability over baselines (e.g., BART) is seen via intrinsic and extrinsic methods, where idiom embeddings score 0.19 points higher in homogeneity score for embedding clustering, and up to 25% higher sequence accuracy on the idiom processing tasks of IE sense disambiguation and span detection. 1 Introduction Natural language has a common yet special class of multi-word expressions (MWEs) called idiomatic expressions (IEs) that exhibit seman- tic non-compositionality, where the meaning of the expression cannot be inferred from that of its constituent words (e.g., the idiom break a leg) (Baldwin and Kim, 2010). They are com- monly used for specific communicative intents (Moon, 1998; Baldwin and Kim, 2010) and are individually rare but collectively frequent, ap- pearing frequently across genres (Moon, 1998; Haagsma et al., 2020). They have been classically regarded as a ‘‘pain in the neck’’ to NLP sys- tems (Sag et al., 2002) not only because of their non-compositionality, but also because of their contextual semantic ambiguity (used in idiomatic or literal meaning depending on the context). Challenges posed by the presence of IEs have been identified across multiple NLU tasks even with state-of-the-art (SOTA) solutions, including sentiment analysis (Liu et al., 2017; Biddle et al., 2020), paraphrase generation (Zhou et al., 2021), natural language inference (Chakrabarty et al., 2021), and dialog models (Jhamtani et al., 2021). Even the the flagship NLP model GPT-3 (Brown et al., 2020) finds idioms challenging. We tested for its idiom comprehension over 75 idioms, covering a spectrum of the most to the least frequent idioms (based on their frequency of occurrence in the BNC (Haagsma et al., 2020)). We do this in question-answering mode where we ask GPT-31 simple questions about idioms that can only be answered correctly if their meanings are understood in context (a sample of questions and answers are shown in Table 1) and noted 37 incorrect answers (50.7% accuracy), showing its inability to comprehend many idioms. Despite the relative success of recent studies us- ing pre-trained language models (PTLMs) for IE processing, including idiom token and type clas- sification (Liu, 2019), IE span detection (Zeng and Bhat, 2021; ˇSkvorc et al., 2022), and id- iomatic to literal paraphrasing (Zhou et al., 2021), the emerging knowledge is that PTLMs are un- able to produce meaningful representation of IEs in task-specific contexts (e.g., for IE sense dis- ambiguation) (Tayyar …

Content truncated to 500 words