IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10660–10668 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10660 INDOBERTWEET: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization Fajri Koto Jey Han Lau Timothy Baldwin School of Computing and Information Systems The University of Melbourne ffajri@student.unimelb.edu.au, jeyhan.lau@gmail.com, tb@ldwin.net Abstract We present INDOBERTWEET, the first large- scale pretrained model for Indonesian Twitter that is trained by extending a monolingually- trained Indonesian BERT model with additive domain-specific vocabulary. We focus in par- ticular on efficient model adaptation under vo- cabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabu- lary adaptation in terms of extrinsic evaluation over seven Twitter-based datasets.1 1 Introduction Transformer-based pretrained language models (Vaswani et al., 2017; Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019) have become the back- bone of modern NLP systems, due to their success across various languages and tasks. However, ob- taining high-quality contextualized representations for specific domains/data sources such as biomedi- cal, social media, and legal, remains a challenge. Previous studies (Alsentzer et al., 2019; Chalkidis et al., 2020; Nguyen et al., 2020) have shown that for domain-specific text, pretraining from scratch outperforms off-the-shelf BERT. As an alternative approach with lower cost, Gururan- gan et al. (2020) demonstrated that domain adap- tive pretraining (i.e. pretraining the model on target domain text before task fine-tuning) is effective, although still not as good as training from scratch. The main drawback of domain-adaptive pretrain- ing is that domain-specific words that are not in the pretrained vocabulary are often tokenized poorly. For instance, in BIOBERT (Lee et al., 2019), Im- munoglobulin is tokenized into {I, ##mm, ##uno, ##g, ##lo, ##bul, ##in}, despite being a common 1Code and models can be accessed at https:// github.com/indolem/IndoBERTweet term in biology. To tackle this problem, Poerner et al. (2020); Tai et al. (2020) proposed simple methods to domain-extend the BERT vocabulary: Poerner et al. (2020) initialize new vocabulary us- ing a learned projection from word2vec (Mikolov et al., 2013), while Tai et al. (2020) use random ini- tialization with weight augmentation, substantially increasing the number of model parameters. New vocabulary augmentation has been also con- ducted for language-adaptive pretraining, mainly based on multilingual BERT (MBERT). For in- stance, Chau et al. (2020) replace 99 “unused” WordPiece tokens of MBERT with new com- mon tokens in the target language, while Wang et al. (2020) extend MBERT vocabulary with non- overlapping tokens (|VMBERT − Vnew|). These two approaches use random initialization for new Word- Piece token embeddings. In this paper, we focus on the task of learning an Indonesian BERT model for Twitter, and show that initializing domain-specific vocabulary with average-pooling of BERT subword embeddings is more efficient than pretraining from scratch, and more effective than initializing based on word2vec projections (Poerner …

Content truncated to 500 words