DagoBERT: Generating Derivational Morphology with a Pretrained Language Model

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3848–3861, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3848 DagoBERT: Generating Derivational Morphology with a Pretrained Language Model Valentin Hofmann*‡, Janet B. Pierrehumbert†*, Hinrich Sch¨utze‡ *Faculty of Linguistics, University of Oxford †Department of Engineering Science, University of Oxford ‡Center for Information and Language Processing, LMU Munich valentin.hofmann@ling-phil.ox.ac.uk Abstract Can pretrained language models (PLMs) gen- erate derivationally complex words? We present the first study investigating this ques- tion, taking BERT as the example PLM. We ex- amine BERT’s derivational capabilities in dif- ferent settings, ranging from using the unmod- ified pretrained model to full finetuning. Our best model, DagoBERT (Derivationally and generatively optimized BERT), clearly outper- forms the previous state of the art in deriva- tion generation (DG). Furthermore, our exper- iments show that the input segmentation cru- cially impacts BERT’s derivational knowledge, suggesting that the performance of PLMs could be further improved if a morphologically informed vocabulary of units were used. 1 Introduction What kind of linguistic knowledge is encoded by pretrained language models (PLMs) such as ELMo (Peters et al., 2018), GPT-2 (Radford et al., 2019), and BERT (Devlin et al., 2019)? This question has attracted a lot of attention in NLP recently, with a focus on syntax (e.g., Goldberg, 2019) and seman- tics (e.g., Ethayarajh, 2019). It is much less clear what PLMs learn about other aspects of language. Here, we present the first study on the knowledge of PLMs about derivational morphology, taking BERT as the example PLM. Given an English cloze sen- tence such as this jacket is . and a base such as wear, we ask: can BERT generate correct derivatives such as unwearable? The motivation for this study is twofold. On the one hand, we add to the growing body of work on the linguistic capabilities of PLMs. Most PLMs segment words into subword units (Bostrom and Durrett, 2020), e.g., unwearable is segmented into un, ##wear, ##able by BERT’s WordPiece tokenizer (Wu et al., 2016). The fact that many of BERT DCL this jacket is [MASK] wear [MASK] . un ##able Figure 1: Basic experimental setup. We input sen- tences such as this jacket is unwearable . to BERT, mask out derivational affixes, and recover them using a derivational classification layer (DCL). Type Examples Prefixes anti, auto, contra, extra, hyper, mega, mini, multi, non, proto, pseudo Suffixes ##able, ##an, ##ate, ##ee, ##ess, ##ful, ##ify, ##ize, ##ment, ##ness, ##ster Table 1: Examples of derivational affixes in the BERT WordPiece vocabulary. Word-internal WordPiece to- kens are marked with ## throughout the paper. these subword units are derivational affixes sug- gests that PLMs might acquire knowledge about derivational morphology (Table 1), but this has not been tested. On the other hand, we are interested in derivation generation (DG) per se, a task that has been only addressed using LSTMs (Cotterell et al., 2017; Vylomova et al., 2017; Deutsch et al., 2018), not models based on Transformers like BERT. Contributions. We develop the first …

Content truncated to 500 words