CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

Proceedings of the 28th International Conference on Computational Linguistics, pages 6903–6915 Barcelona, Spain (Online), December 8-13, 2020 6903 CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters Hicham El Boukkouri1, Olivier Ferret2, Thomas Lavergne1, Hiroshi Noji3, Pierre Zweigenbaum1, Junichi Tsujii3 1Universit´e Paris-Saclay, CNRS, LIMSI, Orsay, France, 2Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France, 3Artificial Intelligence Research Center (AIRC), AIST, Japan {elboukkouri,lavergne,pz}@limsi.fr, olivier.ferret@cea.fr {hiroshi.noji,j-tsujii}@aist.go.jp Abstract Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transform- ers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level, and open-vocabulary representations. 1 Introduction Pre-trained language representations from Transformers (Vaswani et al., 2017) have become arguably the most popular choice for building NLP systems1. Among all such models, BERT (Devlin et al., 2019) has probably been the most successful, spawning a large number of new improved variants (Liu et al., 2019; Lan et al., 2019; Sun et al., 2019; Zhang et al., 2019; Clark et al., 2020). As a result, many of the recent language representation models inherited BERT’s subword tokenization system which relies on a predefined set of wordpieces (Wu et al., 2016), supposedly striking a good balance between the flexibility of characters and the efficiency of full words. While current research mostly focuses on improving language representations for the default “general- domain”, there seems to be a growing interest in building suitable word embeddings for more specialized domains (El Boukkouri et al., 2019; Si et al., 2019; Elwany et al., 2019). However, with the growing complexity of recent representation models, the default trend seems to favor re-training general-domain models on specialized corpora rather than building models from scratch with a specialized vocabulary (e.g., BlueBERT (Peng et al., 2019) and BioBERT (Lee et al., 2020)). While these methods undeniably produce good models 2, a few questions remain: How suitable are the predefined general-domain vocabularies when used in the context of specialized domains (e.g., the medical domain)? Is it better to train specialized models with specialized subword units? Do we induce any biases by training specialized models with general-domain wordpieces? In this paper, we propose CharacterBERT, a possible solution for avoiding any biases that may …

Content truncated to 500 words