Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5617 - 5633 December 7-11, 2022 ©2022 Association for Computational Linguistics Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations Zhihui Xie1 Handong Zhao2 Tong Yu2 Shuai Li1∗ 1Shanghai Jiao Tong University 2Adobe Research {fffffarmer,shuaili8}@sjtu.edu.cn {hazhao,tyu}@adobe.com Abstract Large pretrained multilingual language models (ML-LMs) have shown remarkable capabili- ties of zero-shot cross-lingual transfer, without direct cross-lingual supervision. While these results are promising, follow-up works found that, within the multilingual embedding spaces, there exists strong language identity informa- tion which hinders the expression of linguistic factors shared across languages. For seman- tic tasks like cross-lingual sentence retrieval, it is desired to remove such language identity signals to fully leverage semantic information. In this work, we provide a novel view of pro- jecting away language-specific factors from a multilingual embedding space. Specifically, we discover that there exists a low-rank subspace that primarily encodes information irrelevant to semantics (e.g., syntactic information). To identify this subspace, we present a simple but effective unsupervised method based on singu- lar value decomposition with multiple mono- lingual corpora as input. Once the subspace is found, we can directly project the original em- beddings into the null space to boost language agnosticism without finetuning. We system- atically evaluate our method on various tasks including the challenging language-agnostic QA retrieval task. Empirical results show that applying our method consistently leads to im- provements over commonly used ML-LMs. 1 Introduction Large language models pretrained with self- supervised objectives (e.g., masked language mod- eling) have become the de-facto standard for var- ious NLP tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019). Follow-up extensions to the multilingual setting inherit similar training ob- jectives and show very promising results (Con- neau and Lample, 2019; Conneau et al., 2020b; K et al., 2020). Despite these models are trained ∗Corresponding author. without explicit cross-lingual signals (i.e., trans- lation pairs), they surprisingly exhibit impressive zero-shot cross-lingual transferability on natural language inference (Conneau et al., 2018), ques- tion answering (Lewis et al., 2020), sentence re- trieval (Artetxe and Schwenk, 2019), etc. While these ML-LMs offer practical solutions for cross-lingual tasks, there is an enduring debate about why the ML-LMs work. From a positive per- spective, Pires et al. (2019) conduct an exploratory study on mBERT (Devlin et al., 2019), suggesting that cross-lingual transfer is possible even to lan- guages in different scripts. Chi et al. (2020) probe mBERT for structural phenomena and find that its representations can recover syntactic tree distances in languages other than English. These findings present shreds of evidence that the pretrained mul- tilingual representations do capture cross-lingual properties in various aspects. On the flip side, a line of research shows that pretrained ML-LMs encode strong language-specific signals. This causes their multilingual representations to cluster by language identities instead of semantic meaning (Wu and Dredze, 2019; Roy et al., 2020; Libovický et al., 2020). The property largely hinders the expression of linguistic signals shared across languages. For applications like …

Content truncated to 500 words