Improving Pre-Trained Multilingual Model with Vocabulary Expansion

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 316–327 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 316 Improving Pre-Trained Multilingual Models with Vocabulary Expansion Hai Wang1* Dian Yu2 Kai Sun3* Jianshu Chen2 Dong Yu2 1Toyota Technological Institute at Chicago, Chicago, IL, USA 2Tencent AI Lab, Bellevue, WA, USA 3Cornell, Ithaca, NY, USA haiwang@ttic.edu, ks985@cornell.edu, {yudian,jianshuchen,dyu}@tencent.com Abstract Recently, pre-trained language models have achieved remarkable success in a broad range of natural language processing tasks. How- ever, in multilingual setting, it is extremely resource-consuming to pre-train a deep lan- guage model over large-scale corpora for each language. Instead of exhaustively pre-training monolingual language models independently, an alternative solution is to pre-train a pow- erful multilingual deep language model over large-scale corpora in hundreds of languages. However, the vocabulary size for each lan- guage in such a model is relatively small, es- pecially for low-resource languages. This lim- itation inevitably hinders the performance of these multilingual models on tasks such as se- quence labeling, wherein in-depth token-level or sentence-level understanding is essential. In this paper, inspired by previous methods designed for monolingual settings, we in- vestigate two approaches (i.e., joint mapping and mixture mapping) based on a pre-trained multilingual model BERT for addressing the out-of-vocabulary (OOV) problem on a vari- ety of tasks, including part-of-speech tagging, named entity recognition, machine translation quality estimation, and machine reading com- prehension. Experimental results show that using mixture mapping is more promising. To the best of our knowledge, this is the first work that attempts to address and discuss the OOV issue in multilingual settings. 1 Introduction It has been shown that performance on many natural language processing tasks drops dramati- cally on held-out data when a significant percent- age of words do not appear in the training data, * This work was done when H. W. and K. S. were at Tencent AI Lab, Bellevue, WA. i.e., out-of-vocabulary (OOV) words (Søgaard and Johannsen, 2012; Madhyastha et al., 2016). higher OOV rate (i.e., the percentage of the unseenA words in the held-out data) may lead to a more severe performance drop (Kaljahi et al., 2015). OOV problems have been addressed in previous works under monolingual settings, through replac- ing OOV words with their semantically similar in- vocabulary words (Madhyastha et al., 2016; Ko- lachina et al., 2017) or using character/word infor- mation (Kim et al., 2016, 2018; Chen et al., 2018) or subword information like byte pair encoding (BPE) (Sennrich et al., 2016; Stratos, 2017). Recently, fine-tuning a pre-trained deep lan- guage model, such as Generative Pre-Training (GPT) (Radford et al., 2018) and Bidirec- tional Encoder Representations from Transform- ers (BERT) (Devlin et al., 2018), has achieved re- markable success on various downstream natural language processing tasks. Instead of pre-training many monolingual models like the existing En- glish GPT, English BERT, and Chinese BERT, a more natural choice is to develop a powerful mul- tilingual model such as the multilingual BERT. However, all those pre-trained models rely on language modeling, …

Content truncated to 500 words