Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3418–3430 August 1–6, 2021. ©2021 Association for Computational Linguistics 3418 Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment Zewen Chi†∗, Li Dong‡, Bo Zheng‡∗, Shaohan Huang‡ Xian-Ling Mao†, Heyan Huang†, Furu Wei‡ †Beijing Institute of Technology ‡Microsoft Research {czw,maoxl,hhy63}@bit.edu.cn {lidong1,v-zhebo,shaohanh,fuwei}@microsoft.com Abstract The cross-lingual language models are typi- cally pretrained with masked language model- ing on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-labels word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately per- form the above two steps in an expectation- maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question an- swering, and structured prediction. More- over, the model can serve as a pretrained word aligner, which achieves reasonably low error rates on the alignment benchmarks. The code and pretrained parameters are available at github.com/CZWin32768/XLM-Align. 1 Introduction Despite the current advances in NLP, most applica- tions and resources are still English-centric, mak- ing non-English users hard to access. Therefore, it is essential to build cross-lingual transferable mod- els that can learn from the training data in high- resource languages and generalize on low-resource languages. Recently, pretrained cross-lingual lan- guage models have shown their effectiveness for cross-lingual transfer. By pre-training on monolin- gual text and parallel sentences, the models provide significant improvements on a wide range of cross- lingual end tasks (Conneau and Lample, 2019; Con- neau et al., 2020; Liu et al., 2020; Chi et al., 2021b). Cross-lingual language model pre-training is typ- ically achieved by learning various pretext tasks on ∗Contribution during internship at Microsoft Research. monolingual and parallel corpora. By simply learn- ing masked language modeling (MLM; Devlin et al. 2019) on monolingual text of multiple languages, the models surprisingly achieve competitive results on cross-lingual tasks (Wu and Dredze, 2019; K et al., 2020). Besides, several pretext tasks are proposed to utilize parallel corpora to learn better sentence-level cross-lingual representations (Con- neau and Lample, 2019; Chi et al., 2021b; Hu et al., 2020a). For example, the translation language mod- eling (TLM; Conneau and Lample 2019) task per- forms MLM on the concatenated parallel sentences, which implicitly enhances cross-lingual transfer- ability. However, most pretext tasks either learn alignment at the sentence level or implicitly en- courage cross-lingual alignment, leaving explicit fine-grained alignment task not fully explored. In this paper, we introduce a new cross-lingual pre-training task, named as denoising word align- ment. Rather than relying on external word aligners trained on parallel corpora (Cao et al., 2020; Zhao et al., 2020; Wu and Dredze, 2020), we utilize self-labeled alignments in our task. During pre- training, we alternately self-label word alignments …

Content truncated to 500 words