Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2485–2494, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2485 Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks Haoyang Huang § Yaobo Liang § Nan Duan§ Ming Gong† Linjun Shou† Daxin Jiang† Ming Zhou§ § Microsoft Research Asia, Beijing, China † STCA NLP Group, Microsoft, Beijing, China {haohua,yalia,nanduan,migon,lisho,djiang,mingzhou}@microsoft.com Abstract We present Unicoder, a universal language en- coder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Com- paring to similar efforts such as Multilingual BERT (Devlin et al., 2018) and XLM (Lam- ple and Conneau, 2019), three new cross- lingual pre-training tasks are proposed, in- cluding cross-lingual word recovery, cross- lingual paraphrase classification and cross- lingual masked language model. These tasks help Unicoder learn the mappings among dif- ferent languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improve- ment. Experiments are performed on two tasks: cross-lingual natural language infer- ence (XNLI) and cross-lingual question an- swering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improve- ment (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained. Introduction Data annotation is expensive and time-consuming for most of NLP tasks. Recently, pre-trained mod- els, such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2018) and GPT (Radford et al., 2018), have shown strong capabilities of trans- ferring knowledge learned from large-scale text corpus to specific NLP tasks with limited or no training data. But they still cannot handle tasks when training and test instances are in different languages. Motivated by this issue, some efforts have been made, such as Multilingual BERT (Devlin et al., 2018) and XLM (Lample and Conneau, 2019), for cross-lingual tasks. Multilingual BERT trains a BERT model based on multilingual Wikipedia, which covers 104 languages. As its vocabulary contains tokens from all languages, Multilingual BERT can be used to cross-lingual tasks directly. XLM further improves Multilingual BERT by in- troducing a translation language model (TLM). TLM takes a concatenation of a bilingual sentence pair as input and performs masked language model based on it. By doing this, it learns the mappings among different languages and performs good on the XNLI dataset. However, XLM only uses a single cross-lingual task during pre-training. At the same time, Liu et al. (2019) has shown that multi-task learn- ing can further improve a BERT-style pre-trained model. So we think more cross-lingual tasks could further improve the resulting pre-trained model for cross-lingual tasks. To verify this, we propose Unicoder, a universal language encoder that is insensitive to different languages and pre-trained based on 5 pre-training tasks. Besides masked …

Content truncated to 500 words