mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1671–1683 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1671 mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs Zewen Chi†‡∗, Li Dong‡, Shuming Ma‡, Shaohan Huang‡ Xian-Ling Mao†, Heyan Huang†, Furu Wei‡ †Beijing Institute of Technology ‡Microsoft Research {czw,maoxl,hhy63}@bit.edu.cn {lidong1,shumma,shaohanh,fuwei}@microsoft.com Abstract Multilingual T5 (MT5; Xue et al. 2020) pre- trains a sequence-to-sequence model on mas- sive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text- to-text transfer Transformer with translation pairs (MT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corrup- tion. In addition, we propose a partially non- autoregressive objective for text-to-text pre- training. We evaluate the methods on eight multilingual benchmark datasets, including sentence classification, named entity recogni- tion, question answering, and abstractive sum- marization. Experimental results show that the proposed MT6 improves cross-lingual transfer- ability over MT5. 1 Introduction Multilingual pretrained language models, such as mBERT (Devlin et al., 2019), have attracted in- creasing attention. They not only improve the performance on downstream multilingual NLP tasks (Conneau and Lample, 2019; Conneau et al., 2020; Liu et al., 2020; Chi et al., 2021c), but also show an impressive cross-lingual transferabil- ity (Wu and Dredze, 2019; K et al., 2020; Hu et al., 2020b; Chi et al., 2021a). Multilingual pretrained models are typically trained on multilingual unlabeled text with unsu- pervised language modeling tasks, e.g., masked language modeling (Devlin et al., 2019), causal language modeling (Conneau and Lample, 2019), and span corruption (Raffel et al., 2020). These unsupervised tasks are built upon large-scale mono- lingual texts. In addition, several studies pro- pose cross-lingual tasks that utilize translation data from multilingual parallel corpora, such as trans- lation language modeling (Conneau and Lample, ∗Contribution during internship at Microsoft Research. 2019), cross-lingual contrast (Chi et al., 2021a), and bidirectional word alignment (Hu et al., 2020a). Thanks to the translation data, the pretrained mod- els produce better-aligned cross-lingual representa- tions and obtain better cross-lingual transferability. Recently, the multilingual text-to-text transfer Transformer (MT5; Xue et al. 2020) achieves state- of-the-art performance on several cross-lingual un- derstanding benchmarks. MT5 inherits the benefits of T5 (Raffel et al., 2020) that treats every text pro- cessing problem as a text-to-text problem, i.e., the problem of generating some target text conditioned on the input text. Despite the effectiveness of MT5, how to improve MT5 with translation data is still an open problem. In this paper, we present MT6, standing for improving multilingual text-to-text transfer Trans- former with translation data. MT6 differs from MT5 in terms of both pre-training tasks and the training objective. We present three cross-lingual tasks for text-to-text Transformer pre-training, i.e., machine translation, translation pair span corrup- tion, and translation span corruption. In the trans- lation span corruption task, the model is trained to predict the text spans based on the input translation pair. The cross-lingual tasks encourage the model to align representations of …

Content truncated to 500 words