VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3980–3994 August 1–6, 2021. ©2021 Association for Computational Linguistics 3980 VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation Fuli Luo∗, Wei Wang∗, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, Luo Si Alibaba Group {lfl259702,hebian.ww,glacier.ljh,yanshan.lyj}@alibaba-inc.com {b.bi,songfang.hsf,f.huang,luo.si}@alibaba-inc.com Abstract Existing work in multilingual pretraining has demonstrated the potential of cross-lingual transferability by training a unified Trans- former encoder for multiple languages. How- ever, much of this work only relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages, which is loose and implicit for aligning the contextual representations between languages. In this paper, we plug a cross-attention mod- ule into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. More impor- tantly, when fine-tuning on downstream tasks, the cross-attention module can be plugged in or out on-demand, thus naturally benefiting a wider range of cross-lingual tasks, from lan- guage understanding to generation. As a result, the proposed cross-lingual model delivers new state-of-the-art results on vari- ous cross-lingual understanding tasks of the XTREME benchmark, covering text classifi- cation, sequence labeling, question answer- ing, and sentence retrieval. For cross-lingual generation tasks, it also outperforms all ex- isting cross-lingual models and state-of-the- art Transformer variants on WMT14 English- to-German and English-to-French translation datasets, with gains of up to 1∼2 BLEU. 1 1 Introduction Cross-lingual pre-trained models like mBERT (De- vlin et al., 2019), XLM (Lample and Conneau, 2019) and XLM-R (Conneau et al., 2019) that tar- get providing contextualized representations for the inputs across languages, have shown large poten- *Equal contribution. 1Code and model are available at https://github. com/alibaba/AliceMind/tree/main/VECO 2021/2/1 Lightshot screenshot chrome-extension://mbniclmhobmnbdlbpiphghaielnnpgdp/screenshot.html?id=screenshot_0.0007918474100581108 1/1 (a) XLM (MLM + TLM) (b) XLM-R (MLM) Figure 1: The attention scores of XLM and XLM-R with the input of a pair of parallel sentences: Take a seat and have a rest in English and its translated Chi- nese sentence. The darker line denotes a higher score. We can found that there are only a few attention pat- terns across English and Chinese subwords. tial on a variety of cross-lingual understanding and generation tasks. Behind the great success, two major factors play the role of aligning the contextual representations between languages: 1) build the shared vocabulary across languages through subword tokenization, which supports the simple extension of masked lan- guage modeling (MLM) from English corpus to multilingual corpus; 2) capture the alignment in parallel data via concatenating two sentences as input, called translation language modeling (TLM). However, both of these two mechanisms rely on the self-attention module (query=key/value) of the Transformer encoder to implicitly enhance the in- terdependence between languages, which may lead to few attention patterns across languages. Tak- ing Figure 1 as an example, even though inputting a pair of parallel sentences, both models only at- …

Content truncated to 500 words