Domain-Specific Japanese ELECTRA Model Using a Small Corpus

Proceedings of Recent Advances in Natural Language Processing, pages 640–646 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_072 640 Domain-Specific Japanese ELECTRA Model Using a Small Corpus Youki Itoh and Hiroyuki Shinnou Faculty of Engineering, Ibaraki University 4-12-1 Nakanarusawa, Hitachi, Ibaraki 316-8511 Japan {21nm708a, hiroyuki.shinnou.0828}@vc.ibaraki.ac.jp Abstract Recently, domain shift, which affects accuracy due to differences in data between source and target domains, has become a serious issue when using machine learning methods to solve natural language processing tasks. With addi- tional pretraining and fine-tuning using a tar- get domain corpus, pretraining models such as BERT1 can address this issue. However, the additional pretraining of the BERT model is dif- ficult because it requires significant computing resources. The efficiently learning an encoder that classi- fies token replacements accurately (ELECTRA) pretraining model replaces the BERT pretrain- ing method’s masked language modeling with a method called replaced token detection, which improves the computational efficiency and al- lows the additional pretraining of the model to a practical extent. Herein, we propose a method for addressing the computational efficiency of pretraining models in domain shift by constructing an ELECTRA pretraining model on a Japanese dataset and ad- ditional pretraining this model in a downstream task using a corpus from the target domain. We constructed a pretraining model for ELEC- TRA in Japanese and conducted experiments on a document classification task using data from Japanese news articles. Results show that even a model smaller than the pretrained model performs equally well. 1 Introduction A domain shift problem occurs when using ma- chine learning to solve natural language processing tasks, where the training data (source) domain is differs from the test data (target) domain. Because downstream tasks fine-tune a model, pretrained models such as BERT (Devlin et al., 1Bidirectional Encoder Representations from Transform- ers 2019) can deal with the domain shift problem. To improve accuracy, the additional pretraining of BERT using the target domain corpus and fine- tuning of the additional pretrained models have been used recently. However, the additional pre- training of BERT requires considerable computing resources and therefore cannot be performed easily. Furthermore, a large corpus of the target domain to be used is required, but this corpus is often unavail- able in reality. In this paper, we attempted to address the compu- tational efficiency of pretraining in domain shifting using efficiently learning an encoder that classifies token replacements accurately (ELECTRA) (Clark et al., 2020). ELECTRA is a pretraining model that uses replaced token detection (RTD) to replace the masked language modeling (MLM) used in BERT. In MLM, the model is trained by estimating the [MASK] and replacing some words in the input sentence with [MASK]. However, only 15% of the words in BERT are replaced by [MASK], which is computationally inefficient: thus, RTD is an im- provement on BERT. RTD provides two models: generative and dis- criminative, which are based on the idea of genera- tive adversarial network (GAN) (Goodfellow et al., 2014). The discriminator is pretrained by deciding if it replaces each token generated by the generator. …

Content truncated to 500 words