XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6170 - 6182 May 22-27, 2022 c⃝2022 Association for Computational Linguistics XLM-E: Cross-lingual Language Model Pre-training via ELECTRA Zewen Chi†∗, Shaohan Huang‡∗, Li Dong‡, Shuming Ma‡, Bo Zheng‡, Saksham Singhal‡ Payal Bajaj‡, Xia Song‡, Xian-Ling Mao†, Heyan Huang†, Furu Wei‡ † Beijing Institute of Technology ‡ Microsoft Corporation https://github.com/microsoft/unilm Abstract In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual lan- guage model pre-training. Specifically, we present two pre-training tasks, namely multi- lingual replaced token detection, and transla- tion replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, anal- ysis shows that XLM-E tends to obtain better cross-lingual transferability. 1 Introduction It has become a de facto trend to use a pretrained language model (Devlin et al., 2019; Dong et al., 2019; Yang et al., 2019b; Bao et al., 2020) for downstream NLP tasks. These models are typically pretrained with masked language modeling objec- tives, which learn to generate the masked tokens of an input sentence. In addition to monolingual rep- resentations, the masked language modeling task is effective for learning cross-lingual representations. By only using multilingual corpora, such pretrained models perform well on zero-shot cross-lingual transfer (Devlin et al., 2019; Conneau et al., 2020), i.e., fine-tuning with English training data while di- rectly applying the model to other target languages. The cross-lingual transferability can be further im- proved by introducing external pre-training tasks using parallel corpus, such as translation language modeling (Conneau and Lample, 2019), and cross- lingual contrast (Chi et al., 2021b). However, pre- vious cross-lingual pre-training based on masked language modeling usually requires massive com- putation resources, rendering such models quite expensive. As shown in Figure 1, our proposed ∗ Equal contribution. Zewen Chi contributes during in- ternship at Microsoft Research. 0 20 40 60 80 100 120 FLOPs (1e20) 75.0 75.2 75.4 75.6 75.8 76.0 76.2 76.4 76.6 XNLI Acc XLM-R InfoXLM XLM-Align XLM-R+TLM (0.3M) XLM-R+TLM (0.6M) XLM-R+TLM (0.9M) XLM-R+TLM (1.2M) XLM-R+TLM (1.5M) XLM-E (45K) XLM-E (90K) XLM-E (125K) Speedup 130x Figure 1: The proposed XLM-E pre-training (red line) achieves 130× speedup compared with an in-house pre- trained XLM-R augmented with translation language modeling (XLM-R + TLM; blue line), using the same corpora and code base. The training steps are shown in the brackets. We also present XLM-R (Conneau et al., 2020), InfoXLM (Chi et al., 2021b), and XLM- Align (Chi et al., 2021c). The compared models are all in Base size. XLM-E achieves a huge speedup compared with well-tuned pretrained models. In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two discriminative pre-training tasks, namely multilin- gual replaced token detection, and translation re- placed token detection. Rather than recovering masked tokens, the model learns to distinguish the replaced tokens in …

Content truncated to 500 words