Inducing Language-Agnostic Multilingual Representations

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 229–240 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 229 Inducing Language-Agnostic Multilingual Representations Wei Zhao† Steffen Eger† Johannes BjervaΨ,Φ Isabelle AugensteinΦ †Technische Universit¨at Darmstadt ΦUniversity of Copenhagen ΨAalborg University {zhao,eger}@aiphes.tu-darmstadt.de jbjerva@cs.aau.dk augenstein@di.ku.dk Abstract Cross-lingual representations have the poten- tial to make NLP techniques available to the vast majority of languages in the world. How- ever, they currently require large pretraining corpora or access to typologically similar lan- guages. In this work, we address these ob- stacles by removing language identity signals from multilingual embeddings. We exam- ine three approaches for this: (i) re-aligning the vector spaces of target languages (all to- gether) to a pivot source language; (ii) remov- ing language-specific means and variances, which yields better discriminativeness of em- beddings as a by-product; and (iii) increas- ing input similarity across languages by re- moving morphological contractions and sen- tence reordering. We evaluate on XNLI and reference-free MT across 19 typologically di- verse languages. Our findings expose the limi- tations of these approaches—unlike vector nor- malization, vector space re-alignment and text normalization do not achieve consistent gains across encoders and languages. Due to the ap- proaches’ additive effects, their combination decreases the cross-lingual transfer gap by 8.9 points (m-BERT) and 18.2 points (XLM-R) on average across all tasks and languages, how- ever. Our code and models are publicly avail- able.1 1 Introduction Cross-lingual text representations (Devlin et al., 2019; Conneau et al., 2019) ideally allow for trans- fer between any language pair, and thus hold the promise to alleviate the data sparsity problem for low-resource languages. However, until now, cross- lingual systems trained on English appear to trans- fer poorly to target languages dissimilar to English (Wu and Dredze, 2019; Pires et al., 2019) and for 1https://github.com/AIPHES/ Language-Agnostic-Contextualized-Encoders 0.65 0.70 0.75 0.80 0.85 0.90 0.95 Language Similarity 6 4 2 0 2 4 6 Task Performance RFEval (78.2) XNLI (91.7) 0.0 0.5 1.0 1.5 2.0 2.5 Wikipedia articles (in millions) 2 1 0 1 2 3 Task Performance RFEval (66.9) XNLI (72.8) Figure 1: Zero-shot performance on XNLI and RFE- val vs. language similarity to English (top), and data sizes in Wikipedia (bottom). Each point is a language; brackets give the Pearson correlation of points on the x- and y-axis. Zero-shot performance is based on the last layer of m-BERT and is standardized (zero mean, unit standard deviation) for better comparison. which only small monolingual corpora are available (Conneau et al., 2019; Hu et al., 2020; Lauscher et al., 2020), as illustrated in Fig. 1.2 As a remedy, recent work has suggested to train representations on larger multilingual corpora (Conneau et al., 2019) and, more importantly, to re- align them post-hoc so as to address the deficits of state-of-the-art contextualized encoders which have not seen any parallel data during training (Schuster et al., 2019; Wu and Dredze, 2019; Cao et al., 2020). However, re-mapping (i) can be costly, (ii) requires parallel data on word or …

Content truncated to 500 words