A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5825–5832 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5825 A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations Ziyi Yang1∗, Yinfei Yang2, Daniel Cer2, Eric Darve1 1Stanford University {ziyi.yang,darve}@stanford.edu 2Google Research {yinfeiy,cer}@google.com Abstract Language agnostic and semantic-language in- formation isolation is an emerging research di- rection for multilingual representations mod- els. We explore this problem from a novel an- gle of geometric algebra and semantic space. A simple but highly effective method “Lan- guage Information Removal (LIR)” factors out language identity information from semantic related components in multilingual represen- tations pre-trained on multi-monolingual data. A post-training and model-agnostic method, LIR only uses simple linear operations, e.g. matrix factorization and orthogonal projection. LIR reveals that for weak-alignment multi- lingual systems, the principal components of semantic spaces primarily encodes language identity information. We first evaluate the LIR on a cross-lingual question answer re- trieval task (LAReQA), which requires the strong alignment for the multilingual embed- ding space. Experiment shows that LIR is highly effectively on this task, yielding almost 100% relative improvement in MAP for weak- alignment models. We then evaluate the LIR on Amazon Reviews and XEVAL dataset, with the observation that removing language infor- mation is able to improve the cross-lingual transfer performance. 1 Introduction Recently, large-scale language modeling has ex- panded from English to the multilingual setting (i.a., Devlin et al. (2019); Conneau and Lample (2019); Conneau et al. (2020)). Although these models are trained with language modeling ob- jectives on monolingual data, i.e. without cross- lingual information, these multilingual systems ex- hibit impressive zero-shot cross-lingual ability (Hu et al., 2020b). These observations raise many ques- tions and provide insight for multilingual repre- sentations learning. First, how is the language ∗Work done during internship at Google Research. identity information and the semantic information expressed in the representation? Understanding their relations and underlying geometric structure is crucial for insights into designing more effec- tive multilingual embedding systems. Second, how can we factor out the language identity information from the semantic components in representations? In many application, e.g. cross-lingual semantic retrieval, we wish to only keep the semantic in- formation. Third, what is the geometric relation between different languages? Efforts have been made to answer these questions, e.g. Artetxe et al. (2020); Chung et al. (2020); Lauscher et al. (2020). Such prior work has addressed the problem at train- ing time. In this work, we systematically explore a post-training method that can be readily applied to existing multilingual models. One of the first attempts in this research area, Roy et al. (2020), proposed two concepts for lan- guage agnostic models: weak alignment v.s. strong alignment. For multilingual systems with weak alignment, for any item in language L1, the nearest neighbor in language L2 is the most semantically “relevant” item. In the case of strong alignment, for any representation, all semantically relevant items are closer than all irrelevant items, regardless of their language. …

Content truncated to 500 words