BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3996–4007 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3996 BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance Timo Schick Sulzer GmbH Munich, Germany timo.schick@sulzer.de Hinrich Sch¨utze Center for Information and Language Processing LMU Munich, Germany inquiries@cislmu.org Abstract Pretraining deep language models has led to large performance gains in NLP. Despite this success, Schick and Sch¨utze (2020) recently showed that these models struggle to under- stand rare words. For static word embeddings, this problem has been addressed by separately learning representations for rare words. In this work, we transfer this idea to pretrained language models: We introduce BERTRAM, a powerful architecture based on BERT that is capable of inferring high-quality embeddings for rare words that are suitable as input rep- resentations for deep language models. This is achieved by enabling the surface form and con- texts of a word to interact with each other in a deep architecture. Integrating BERTRAM into BERT leads to large performance increases due to improved representations of rare and medium frequency words on both a rare word probing task and three downstream tasks.1 1 Introduction As word embedding algorithms (e.g. Mikolov et al., 2013) are known to struggle with rare words, sev- eral techniques for improving their representations have been proposed. These approaches exploit ei- ther the contexts in which rare words occur (Lazari- dou et al., 2017; Herbelot and Baroni, 2017; Kho- dak et al., 2018; Liu et al., 2019a), their surface- form (Luong et al., 2013; Bojanowski et al., 2017; Pinter et al., 2017), or both (Schick and Sch¨utze, 2019a,b; Hautte et al., 2019). However, all of this prior work is designed for and evaluated on uncon- textualized word embeddings. Contextualized representations obtained from pretrained deep language models (e.g. Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019b) already handle rare words implicitly 1Our implementation of BERTRAM is publicly available at https://github.com/timoschick/bertram. using methods such as byte-pair encoding (Sen- nrich et al., 2016), WordPiece embeddings (Wu et al., 2016) and character-level CNNs (Baevski et al., 2019). Nevertheless, Schick and Sch¨utze (2020) recently showed that BERT’s (Devlin et al., 2019) performance on a rare word probing task can be significantly improved by explicitly learning rep- resentations of rare words using Attentive Mimick- ing (AM) (Schick and Sch¨utze, 2019a). However, AM is limited in two important respects: • For processing contexts, it uses a simple bag- of-words model, making poor use of the avail- able information. • It combines form and context in a shallow fashion, preventing both input signals from interacting in a complex manner. These limitations apply not only to AM, but to all previous work on obtaining representations for rare words by leveraging form and context. While using bag-of-words models is a reasonable choice for static embeddings, which are often themselves bag- of-words (e.g. Mikolov et al., 2013; Bojanowski et al., 2017), it stands to reason that …

Content truncated to 500 words