Multi-Granularity Contrasting for Cross-Lingual Pre-Training

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1708–1717 August 1–6, 2021. ©2021 Association for Computational Linguistics 1708 Multi-Granularity Contrasting for Cross-Lingual Pre-Training Shicheng Li1, Pengcheng Yang2, Fuli Luo2, Jun Xie2 1MOE Key Lab of Computational Linguistics, School of EECS, Peking University 2Alibaba DAMO Academy, Hangzhou, China lisc99@pku.edu.cn {mingyang.ypc, lfl259702, qingjing.xj}@alibaba-inc.com Abstract Cross-lingual pre-training aims at providing effective prior representations for the inputs from multiple languages. With the model- ing of bidirectional contexts, recently preva- lent language modeling approaches such as XLM achieve better performance than tradi- tional methods based on embedding align- ment, which strives to assign similar vector representations to semantic-equivalent units. However, such approaches like XLM cap- ture cross-lingual information based solely on shared BPE vocabulary, resulting in the absence of fine-grained supervision induced by embedding alignment. Inheriting the ad- vantages of the above two paradigms, this work presents a multi-granularity contrasting framework, namely MGC, to learn language- universal representations. While predicting the masked words based on bidirectional con- texts, the proposal also encodes semantic equivalents from different languages into sim- ilar representations to introduce more fine- grained and explicit cross-lingual information. Two effective contrasting strategies are further proposed, which can be built upon semantic units of multiple granularities covering words, span, and sentences. Extensive experiments demonstrate that our approach can achieve sig- nificant performance gains in various down- stream tasks, including machine translation and cross-lingual language understanding. 1 Introduction Cross-lingual pre-training (Lample and Conneau, 2019) has achieved striking success in the field of natural language processing. By providing ef- fective prior representations for the inputs from different languages, it has boosted performance on various downstream tasks such as machine transla- tion and cross-lingual language understanding. Early efforts regarding cross-lingual pre-training mainly focus on embedding alignment (Mikolov et al., 2013b; Lample et al., 2018), which is tar- geted at the assignment of similar vector repre- sentations to semantic-equivalent units (e.g., the parallel bilingual word or sentence pairs). For in- stance, Mikolov et al. (2013b) attempt to project pre-trained monolingual word embeddings from two languages into a common semantic space with a simple linear transformation, so that parallel bilin- gual words share the same representation. This al- lows the introduction of explicit fine-grained super- vision to guarantee the representational similarity of semantic equivalents, but neglects the modeling of bidirectional contexts. Going a step further, re- cently prevalent approaches of language modeling such as XLM (Lample and Conneau, 2019) remedy this by predicting the masked tokens based on bidi- rectional contexts (Devlin et al., 2019), and also benefit from larger model capacity (Vaswani et al., 2017). However, the cross-lingual information captured by these language modeling approaches comes solely from the shared BPE vocabulary (Sen- nrich et al., 2016), resulting in the absence of more fine-grained explicit supervision induced by em- bedding alignment. In light of the pros and cons of the above two paradigms, we propose a multi-granularity con- trasting (MGC) framework for cross-lingual pre- training. In addition to modeling context bidirec- tionality with the widely used …

Content truncated to 500 words