Translation Artifacts in Cross-lingual Transfer Learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7674–7684, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7674 Translation Artifacts in Cross-lingual Transfer Learning Mikel Artetxe, Gorka Labaka, Eneko Agirre HiTZ Center University of the Basque Country (UPV/EHU) {mikel.artetxe,gorka.labaka,e.agirre}@ehu.eus Abstract Both human and machine translation play a central role in cross-lingual transfer learning: many multilingual datasets have been created through professional translation services, and using machine translation to translate either the test set or the training set is a widely used transfer technique. In this paper, we show that such translation process can introduce subtle artifacts that have a notable impact in existing cross-lingual models. For instance, in natu- ral language inference, translating the premise and the hypothesis independently can reduce the lexical overlap between them, which cur- rent models are highly sensitive to. We show that some previous findings in cross-lingual transfer learning need to be reconsidered in the light of this phenomenon. Based on the gained insights, we also improve the state-of-the-art in XNLI for the translate-test and zero-shot ap- proaches by 4.3 and 2.8 points, respectively. 1 Introduction While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks. One possibility is to collect and annotate data in multiple languages separately (Clark et al., 2020), but most exist- ing datasets have been created through translation (Conneau et al., 2018; Artetxe et al., 2020). This ap- proach has two desirable properties: it relies on ex- isting professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a mean- ingful measure of the cross-lingual transfer gap of different models. The resulting multilingual datasets are generally used for evaluation only, re- lying on existing English datasets for training. Closely related to that, cross-lingual transfer learning aims to leverage large datasets avail- able in one language—typically English—to build multilingual models that can generalize to other languages. Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual En- glish model (TRANSLATE-TEST), machine translat- ing the training set into each target language and training the models on their respective languages (TRANSLATE-TRAIN), or using English data to fine- tune a multilingual model that is then transferred to the rest of languages (ZERO-SHOT). The dataset creation and transfer procedures de- scribed above result in a mixture of original,1 human translated and machine translated data when dealing with cross-lingual models. In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: TRANSLATE-TEST systems are trained on original data and evaluated on machine translated test sets, ZERO-SHOT systems are trained on orig- inal data and evaluated on human translated test sets, and TRANSLATE-TRAIN systems are trained on machine translated data and evaluated on human translated test sets. Despite overlooked to date, we show that such mismatch has a notable impact in the …

Content truncated to 500 words