InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576–3588 June 6–11, 2021. ©2021 Association for Computational Linguistics 3576 INFOXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training Zewen Chi†‡∗, Li Dong‡, Furu Wei‡, Nan Yang‡, Saksham Singhal‡, Wenhui Wang‡ Xia Song‡, Xian-Ling Mao†, Heyan Huang†, Ming Zhou‡ †Beijing Institute of Technology ‡Microsoft Corporation {czw,maoxl,hhy63}@bit.edu.cn {lidong1,fuwei,nanya,saksingh,wenwan,xiaso}@microsoft.com Abstract In this work, we present an information- theoretic framework that formulates cross- lingual language model pre-training as maximizing mutual information multilingual-multi-granularity texts.between The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre- training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm. 1 Introduction Learning cross-lingual language representations plays an important role in overcoming the language barrier of NLP models. The recent success of cross- lingual language model pre-training (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020a; Chi et al., 2020; Liu et al., 2020) signif- icantly improves the cross-lingual transferability in various downstream tasks, such as cross-lingual classification, and question answering. State-of-the-art cross-lingual pre-trained mod- els are typically built upon multilingual masked language modeling (MMLM; Devlin et al. 2019; Conneau et al. 2020a), and translation language modeling (TLM; Conneau and Lample 2019). The goal of both pretext tasks is to predict masked to- kens given input context. The difference is that ∗ Contribution during internship at Microsoft Research. Contact person: Li Dong and Furu Wei. MMLM uses monolingual text as input, while TLM feeds bilingual parallel sentences into the model. Even without explicit encouragement of learning universal representations across languages, the de- rived models have shown promising abilities of cross-lingual transfer. In this work, we formulate cross-lingual pre- training from a unified information-theoretic per- spective. Following the mutual information maxi- mization principle (Hjelm et al., 2019; Kong et al., 2020), we show that the existing pretext tasks can be viewed as maximizing the lower bounds of mutual information between various multilingual- multi-granularity views. Specifically, MMLM maximizes mutual infor- mation between the masked tokens and the con- text in the same language while the anchor points across languages encourages the correlation be- tween cross-lingual contexts. Moreover, we present that TLM can maximize mutual information be- tween the masked tokens and the parallel context, which implicitly aligns encoded representations of different languages. The unified information- theoretic framework also inspires us to propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCO). The model learns to distinguish the translation of an input sentence from a set of …

Content truncated to 500 words