Pretraining Language- and Domain-Specific BERT on Automatically Translated Text

Proceedings of Recent Advances in Natural Language Processing, pages 548–555 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_060 548 Pretraining Language- and Domain-Specific BERT on Automatically Translated Text Tatsuya Ishigaki† Yui Uehara†‡ Goran Topi´c† Hiroya Takamura† †National Institute of Advanced Industrial Science and Technology, Japan, ‡Kanagawa University, {ishigaki.tatsuya, goran.topic, takamura.hiroya}@aist.go.jp yuiuehara@kanagawa-u.ac.jp Abstract Domain-specific pretrained language models such as SciBERT are effective for various tasks involving text in specific domains. However, pretraining BERT requires a large-scale lan- guage resource, which is not necessarily avail- able in fine-grained domains, especially in non- English languages. In this study, we focus on a setting with no available domain-specific text for pretraining. To this end, we propose a sim- ple framework that trains a BERT on text in the target language automatically translated from a resource-rich language, e.g., English. In this paper, we particularly focus on the materials science domain in Japanese. Our experiments pertain to the task of entity and relation extrac- tion for this domain and language. The experi- ments demonstrate that the various models pre- trained on translated texts consistently perform better than the general BERT in terms of F1 scores although the domain-specific BERTs do not use any human-authored domain-specific text. These results imply that BERTs for vari- ous low-resource domains can be successfully trained on texts automatically translated from resource-rich languages. 1 Introduction Domain-specific pretrained language models (LMs), such as SciBERT (Beltagy et al., 2019), are known to perform better on many downstream tasks with texts in the specific domain, such as named entity recognition in biomedical (Li et al., 2016) and relation extraction in chemical do- mains (Kringelum et al., 2016). This trend has mo- tivated researchers to release many domain-specific LMs for resource-rich domains and languages, specifically in medicine (Alsentzer et al., 2019), biomedicine (Lee et al., 2019), finance (Araci, 2019), and materials science (Gupta et al., 2021). Many of the domain-specific LMs have been trained on corpora consisting of academic papers Figure 1: Framework for pretraining that uses language- and domain-specific texts obtained through machine translation. or articles, which are usually open to the public. However, such open corpora are often not avail- able in non-English languages. Meanwhile, there are a lot of documents that are not open such as internal corporate documents also in non-English languages, which still need to be processed with pretrained LMs. We focus on a novel setup for pretraining domain-specific BERTs without the use of human- authored domain-specific text. As a solution to the problem, we pretrain LMs on domain-specific text automatically translated from a resource-rich lan- guage, i.e., English. As shown in Figure 1, journal papers are automatically translated from English to the target language, e.g., Japanese in this pa- per, then used in BERT pretraining in different configurations with or without general texts, e.g., Japanese Wikipedia, to investigate the viability on domain-specific Japanese text. Although this is a very simple approach with wide applicability to various domains and languages, the following two questions still need to be answered: 1) is the use of …

Content truncated to 500 words