PatchBERT: Just-in-Time, Out-of-Vocabulary Patching

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7846–7852, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7846 PatchBERT: Just-in-Time, Out-of-Vocabulary Patching Sangwhan Moon Tokyo Institute of Technology Odd Concepts Inc. Tokyo, Japan sangwhan@iki.fi Naoaki Okazaki Tokyo Institute of Technology Tokyo, Japan okazaki@c.titech.ac.jp Abstract Large scale pre-trained language models have shown groundbreaking performance improve- ments for transfer learning in the domain of natural language processing. In our paper, we study a pre-trained multilingual BERT model and analyze the OOV rate on downstream tasks, how it introduces information loss, and as a side-effect, obstructs the potential of the underlying model. We then propose multi- ple approaches for mitigation and demonstrate that it improves performance with the same parameter count when combined with fine- tuning. 1 Background The advent of large scale language models pre- trained with large unannotated corpora has shown significant advancements in the domain of natu- ral language processing, especially by demonstrat- ing their effectiveness through transfer learning for downstream tasks (Howard and Ruder, 2018; Devlin et al., 2019; Conneau and Lample, 2019; Radford et al.), analogous to ImageNet (Deng et al., 2009) pre-trained backbones in the domain of com- puter vision. In the domain of natural language processing, new methods have made it possible to use internet corpora as a nearly free source for in- creasing the amount of data at an unprecedented scale during pre-training. Additionally, new tokenization methods such as Byte-Pair Encoding (BPE, Sennrich et al. (2016)), WordPiece (Wu et al., 2016), SentencePiece (Kudo and Richardson, 2018), which break the lexicons into smaller subwords, have shown to be effec- tive when applied to alphabetic languages to re- duce the size of the vocabulary while increasing the robustness against out-of-vocabulary (OOV) in downstream tasks. This is especially powerful when combined with transfer learning. However, these tokenizers still operate at Unicode charac- ter levels - contrary to the names suggesting byte- level (which would completely mitigate OOV, as studied in Gillick et al. (2016)). Hence, the min- imum size of the vocabulary is twice the size of all unique characters in the corpus, as subword tokenizers store each character in prefix and suf- fix form in the vocabulary. As OOV was a much more prevalent problem in the context of lexicon- based methods, there have been many methods, such as dictionary-based postprocessing (Luong et al., 2015) and distributional representation based substitution (Kolachina et al., 2017). Recently this has not been as actively studied in the context of subword tokenization as Latin languages are no longer affected. For these reasons, when trained against a diverse set of languages, the vocabulary size increases pro- portionally to the number of languages supported. Existing models have sampled portions of entire corpora or relaxed constraints on character level coverage for these languages, to prevent the vo- cabulary from growing to an unmanageable scale. As of today, this is an unavoidable trade-off when training multilingual models. This introduces a bot- tleneck for downstream tasks since any character omitted causes …

Content truncated to 500 words