Construction of Japanese BERT with Fixed Token Embeddings

Construction of Japanese BERT with Fixed Token Embeddings Arata Suganami , Hiroyuki Shinnou Ibaraki University, Department of Computer and Information Sciences 4-12-1 Nakanarusawa, Hitachi, Ibaraki, Japan 316-8511 {22nm721n, hiroyuki.shinnou.0828}@vc.ibaraki.ac.jp Abstract In this study, we propose a method to construct Japanese BERT using fixed Token Embedding to reduce BERT model construction time. This method constructs word embeddings in advance using word2vec and the corpus, which is used in learning BERT models. The token embeddings in the BERT model are set to the constructed word embeddings. The parameters of token embeddings are fixed, that is, not learned during BERT model learning. Therefore, the proposed method reduces the processing time of one epoch in BERT model training and obtains the BERT model in fewer epochs. In the experiments, we construct 1024- dimensional 4-layer and 128-dimensional 24- layer Japanese BERT model using conventional and proposed methods, respectively, and the effectiveness of the proposed method was verified. 1 Introduction BERT (Devlin et al., 2019) is a high-performance pre-trained model, but its large model size requires significant time and computational resources to build. Another problem is the increase in model construction time and cost because of the huge data sets and models that come with the higher performance of pre-trained models (Sharir et al., 2020). To reduce the construction time of BERT, we propose a method to construct Japanese BERT by fixing the word distribution representation. Specifically, we constructed Japanese BERT by learning word embeddings in advance using word2vec (Mikolov et al., 2013) and fixing them as the Token Embedding of BERT. This reduces the time required to learn word embeddings during BERT construction. In the experiments, we constructed 128- dimensional 24-layer and 1024-dimensional 4- layer Japanese BERTs using conventional and proposed methods, respectively, and the effectiveness of the proposed method was verified by comparing the model construction time and accuracy in a document classification task for Japanese news articles. The results showed that the proposed method reduced the construction time for both BERTs. The results of the document classification task showed that the proposed method can learn a model with the same level of accuracy using fewer epochs. 2 Related Work 2.1 Small size BERT BERT is a huge model, and its training requires a significant amount of computation time and computer resources. The amount of time required to process inference using BERT is also significant. To address this problem, many studies have been conducted to miniaturize BERT. DistilBERT (Sanh et al., 2019) is a miniaturized BERT built using distillation (Hinton et al., 2019). ALBERT (Lan et al., 2019) reduced the number of parameters of BERT using two methods, such as Cross-Layer Parameter Sharing and Factorized Embedding Parameterization. Cross-Layer Parameter Sharing is a method to used for sharing the parameters of each layer of BERT. Factorized Embedding Parameterization decomposes a matrix of size VH representing the word embedding into a product of a matrix of size VE and a matrix of size EH, where E is a small number (e.g. 128), the number of parameters …

Content truncated to 500 words