Vocabulary expansion of compound words for domain adaptation of BERT

Vocabulary expansion of compound words for domain adaptation of BERT Hirotaka Tanaka Department of Computer and Information Sciences Ibaraki University Hitachi, JAPAN { 22nd304a, hiroyuki.shinnou.0828 } @vc.ibaraki.ac.jp Hiroyuki Shinnou Abstract Pretraining models such as BERT, have achieved high accuracy in various natu- ral language processing tasks by pretrain- ing on a large corpus and fine-tuning on downstream task data. However, BERT trains token-level inferences, which make it difficult to train unknown or compound words that are split by byte-pair encod- ing. In this paper, we propose an effective method for constructing word representa- tions in vocabulary expansions for such compound words. The proposed method assumes domain adaptation by additional pretraining and expands the vocabulary by embedding a synonym as an approximate embedding of additional words. We con- ducted experiments using each vocabulary expansion method and evaluated these ex- periments for their accuracies in predicting additional vocabularies in the masked lan- guage model. 1 Introduction Pre-learning models have significantly improved the performances of various natural language pro- cessing systems (Peters et al., 2018)(Radford et al., 2018). Bidirectional encoder representations from transformers (BERT)(Devlin et al., 2019) is pretrained model that consists of a stacked multi- head attention in the Transformer(Vaswani et al., 2017). A BERT outputs word representations that embed the context of the input word sequence. Pre-learning models have domain adaptation problems, and they perform various downstream tasks with high accuracy by applying models pre- trained on a large corpus to the downstream task (a) BERT trains Token- by-Token predictions. (b) Token-by-token pre- diction is high perfor- mance. (c) Difficult to predict the entire splitted repre- sentation. (d) The entire represen- tation as a single Token is easily predicted. Figure 1: Examples of vocabulary expansion tar- gets data. Therefore, if the domain of the down- stream task differs significantly from the domain of the pretrained corpus, the solution of the down- stream task will have low accuracy. Gururangan et al.(2020) proposed a method for domain adap- tation by additional pretraining on a corpus of downstream task domains. Domain adaptation problems also appear in the vocabulary. The vocabulary covered by the pre- trained model depends on the pretrained corpus. Therefore, it must adapt to the vocabulary appear- ing in the downstream task data through vocabu- lary expansion. The adapt-and-distill approach by Figure 2: Construction of BERT input embeddings Yao et al.(2021) and AVocaDo strategy by Hong et al.(2021) are vocabulary expansion methods for domain adaptation in the pretraining models. BERT trains token-level expressions, and it has difficulty training the expressions of compound words, named entities, and phrases. The Japanese vocabulary for a standard BERT model is token units, and tokenization is performed by morpho- logical analysis and byte-pair encoding. The divi- sion of unknown words into known words makes it possible to cover a large number of words us- ing a small vocabulary. However, the masked lan- guage model (MLM) in the pretraining task trains for token by token prediction. Therefore, it is difficult to predict an entire representation con- structed using …

Content truncated to 500 words