Will it Unblend?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1525–1535 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1525 Will it Unblend? Yuval Pinter School of Interactive Computing Georgia Institute of Technology Atlanta, GA, USA uvp@gatech.edu Cassandra L. Jacobs Department of Psychology University of Wisconsin Madison, WI, USA cjacobs2@wisc.edu Jacob Eisenstein Google Research Seattle, WA, USA jeisenstein@google.com Abstract Natural language processing systems often struggle with out-of-vocabulary (OOV) terms, which do not appear in training data. Blends, such as innoventor, are one particularly chal- lenging class of OOV, as they are formed by fusing together two or more bases that relate to the intended meaning in unpredictable man- ners and degrees. In this work, we run ex- periments on a novel dataset of English OOV blends to quantify the difficulty of interpret- ing the meanings of blends by large-scale con- textual language models such as BERT. We first show that BERT’s processing of these blends does not fully access the component meanings, leaving their contextual represen- tations semantically impoverished. We find this is mostly due to the loss of characters re- sulting from blend formation. Then, we as- sess how easily different models can recognize the structure and recover the origin of blends, and find that context-aware embedding sys- tems outperform character-level and context- free embeddings, although their results are still far from satisfactory. 1 Introduction For the token-based architectures that dominate contemporary natural language processing, a par- ticularly difficult form of linguistic generaliza- tion arises from unseen phenomena at the word level, where novel sequences of characters, mor- phemes, or phonemes are known as out-of- vocabulary (OOV) terms (Brants, 2000; Plank, 2016; Heigold et al., 2017). Pretrained trans- formers like BERT (Devlin et al., 2019) han- dle OOV terms by subtokenization: segment- ing all whitespace-delimited tokens into smaller units, from which any OOV term can be con- structed (Sennrich et al., 2016).1 But while this ap- 1Another approach is to operate directly at the character level (e.g., Ling et al., 2015), but this has not been widely proach is well suited for phenomena like concate- native English morphology, many linguistic pro- cesses generate OOV terms that cannot be cleanly decomposed into meaningful subtoken segments. In this paper we address a particularly interest- ing and challenging source of OOV terms: novel blends (Algeo, 1977), also known as portman- teaux (Deri and Knight, 2015). Blends are con- structed from the combination of multiple bases into a new form, in which some characters is shared across both bases: for example, shop + op- tics = shoptics. In this way, blends differ from other lexical compounds (e.g., watermelon = wa- ter + melon), which are formed by simple con- catenation. Examples of OOV blends and their bases from our novel English blends dataset, col- lected from a natural source linked to the blends’ originating contexts (§2), are presented in Table 1. OOV blends are especially challenging to process, due to their combination of function-level seman- tic novelty with the form-level pathology …

Content truncated to 500 words