Do Explicit Alignments Robustly Improve Multilingual Encoders?

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4471–4482, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4471 Do Explicit Alignments Robustly Improve Multilingual Encoders? Shijie Wu and Mark Dredze Department of Computer Science Johns Hopkins University shijie.wu@jhu.edu, mdredze@cs.jhu.edu Abstract Multilingual BERT (Devlin et al., 2019, mBERT), XLM-RoBERTa (Conneau et al., 2019, XLMR) and other unsupervised multi- lingual encoders can effectively learn cross- lingual representation. Explicit alignment ob- jectives based on bitexts like Europarl or Mul- tiUN have been shown to further improve these representations. However, word-level alignments are often suboptimal and such bi- texts are unavailable for many languages. In this paper, we propose a new contrastive align- ment objective that can better utilize such signal, and examine whether these previous alignment methods can be adapted to noisier sources of aligned data: a randomly sampled 1 million pair subset of the OPUS collection. Additionally, rather than report results on a single dataset with a single model run, we re- port the mean and standard derivation of multi- ple runs with different seeds, on four datasets and tasks. Our more extensive analysis finds that, while our new objective outperforms pre- vious work, overall these methods do not im- prove performance with a more robust evalua- tion framework. Furthermore, the gains from using a better underlying model eclipse any benefits from alignment training. These neg- ative results dictate more care in evaluating these methods and suggest limitations in apply- ing explicit alignment objectives. 1 Introduction Unsupervised massively multilingual encoders in- cluding multilingual BERT (Devlin et al., 2019, mBERT) and XLM-RoBERTa (Conneau et al., 2019, XLMR) are now standard tools for zero- shot cross-lingual transfer for NLP tasks (Wu and Dredze, 2019; Xia et al., 2020). While almost all encoders are pretrained without explicit cross- lingual objective, i.e. enforcing similar words from Code is available at https://github.com/ shijie-wu/crosslingual-nlp. different languages have similar representation, im- provements can be attained through the use of ex- plicit cross-lingually linked data during pretraining, such as bitexts (Conneau and Lample, 2019; Huang et al., 2019; Ji et al., 2019) and dictionaries (Wu et al., 2019). As with cross-lingual embeddings (Ruder et al., 2019), these data can be used to sup- port explicit alignment objectives with either linear mappings (Wang et al., 2019, 2020; Wu et al., 2019; Liu et al., 2019) or fine-tuning (Cao et al., 2020). However, as word-level alignments from an un- supervised aligner are often suboptimal, we de- velop a new cross-lingual alignment objective for training our model. We base on our objective on contrastive learning, in which two similar inputs – such as from a bitext – are directly optimized to be similar, relative to a negative set. These meth- ods have been effective in computer vision tasks (He et al., 2019; Chen et al., 2020a). Addition- ally, most previous work on contextual alignments consider high-quality bitext like Europarl (Koehn, 2005) or MultiUN (Eisele and Chen, 2010). While helpful, these resources are unavailable for most languages for which …

Content truncated to 500 words