Domain Mismatch Doesn’t Always Prevent Cross-lingual Transfer Learning

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 892–899 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 892 Domain Mismatch Doesn’t Always Prevent Cross-Lingual Transfer Learning Daniel Edmiston,1 Phillip Keung,1 Noah A. Smith2,3 1Amazon, 2University of Washington, 3Allen Institute for Artificial Intelligence danedm@amazon.com, keung@amazon.com, nasmith@cs.washington.edu Abstract Cross-lingual transfer learning without labeled target language data or parallel text has been surprisingly effective in zero-shot cross-lingual classification, question answering, unsupervised machine translation, etc. However, some recent publications have claimed that domain mismatch prevents cross-lingual transfer, and their results show that unsupervised bilingual lexicon induction (UBLI) and unsupervised neural machine translation (UNMT) do not work well when the underlying monolingual corpora come from different domains (e.g., French text from Wikipedia but English text from UN proceedings). In this work, we show that a simple initialization regimen can overcome much of the effect of domain mismatch in cross-lingual transfer. We pre-train word and contextual embeddings on the concatenated domain-mismatched corpora, and use these as initializations for three tasks: MUSE UBLI, UN Parallel UNMT, and the SemEval 2017 cross-lingual word similarity task. In all cases, our results challenge the conclusions of prior work by showing that proper initialization can recover a large portion of the losses incurred by domain mismatch. Keywords: Domain mismatch, cross-lingual transfer, transfer learning, machine translation 1. Introduction Zero-shot cross-lingual transfer via representation learning has been studied in many recent works span- ning a variety of tasks: cross-lingual text classification and named entity recognition (Devlin et al., 2018), un- supervised neural machine translation (Lample et al., 2018a; Artetxe et al., 2018b) and unsupervised bilin- gual lexicon induction (Conneau et al., 2018; Zhang et al., 2017), among others. Cross-lingual transfer tech- niques typically assume that the source and target text come from the same domain (e.g., English and French Wikipedia for UNMT), but many recent papers have reported issues in the domain-mismatched setting (e.g., English Europarl and French Wikipedia). Particularly, domain mismatch has been shown to have a pernicious effect on UBLI, and has been labeled a “core limitation” (Søgaard et al., 2018), with word em- beddings pre-trained on domain-mismatched corpora showing markedly degraded scores. In the case of UNMT, mismatched domains between source and tar- get training data have also been shown to cause large reductions in BLEU scores (Marchisio et al., 2020). The results in Table 1 illustrate the severity of the prob- lem. In this work, we show that cross-lingual transfer can occur even when there is no overlap between the do- mains in the same language and no overlap between the languages in the same domain (Figure 1). Earlier work such as mBERT (Devlin et al., 2018) and XLM (Lam- ple and Conneau, 2019) demonstrated that pre-training contextual embeddings on concatenated multilingual Wikipedia text induces cross-lingual transfer effects. We extend these findings to the domain-mismatched case, where we pre-train our embeddings on concate- Language A Language B Domain 1 Domain 2 Figure 1: In our zero-shot experiments, training cor- …

Content truncated to 500 words