The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2377–2390, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2377 The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures Haim Dubossarsky1 Ivan Vuli´c1 Roi Reichart2 Anna Korhonen1 1 Language Technology Lab, University of Cambridge 2 Faculty of Industrial Engineering and Management, Technion, IIT {hd423, iv250, alk23}@cam.ac.uk roiri@ie.technion.ac.il Abstract Performance in cross-lingual NLP tasks is im- pacted by the (dis)similarity of languages at hand: e.g., previous work has suggested there is a connection between the expected success of bilingual lexicon induction (BLI) and the assumption of (approximate) isomorphism be- tween monolingual embedding spaces. In this work we present a large-scale study focused on the correlations between monolingual em- bedding space similarity and task performance, covering thousands of language pairs and four different tasks: BLI, parsing, POS tagging and MT. We hypothesize that statistics of the spec- trum of each monolingual embedding space indicate how well they can be aligned. We then introduce several isomorphism measures between two embedding spaces, based on the relevant statistics of their individual spectra. We empirically show that 1) language simi- larity scores derived from such spectral iso- morphism measures are strongly associated with performance observed in different cross- lingual tasks, and 2) our spectral-based mea- sures consistently outperform previous stan- dard isomorphism measures, while being com- putationally more tractable and easier to inter- pret. Finally, our measures capture comple- mentary information to typologically driven language distance measures, and the combina- tion of measures from the two families yields even higher task performance correlations. 1 Introduction The effectiveness of joint multilingual modeling and cross-lingual transfer in cross-lingual NLP is critically impacted by the actual languages in consideration (Bender, 2011; Ponti et al., 2019). Characterizing, measuring, and understanding this cross-language variation is often the first step to- wards the development of more robust multilin- gually applicable NLP technology (O’Horan et al., 2016; Bjerva et al., 2019; Ponti et al., 2019). For instance, selecting suitable source languages is a prerequisite for successful cross-lingual transfer of dependency parsers or POS taggers (Naseem et al., 2012; Ponti et al., 2018; de Lhoneux et al., 2018). In another example, with all other factors kept sim- ilar (e.g., training data size, domain similarity), the quality of machine translation also depends heavily on the properties and language proximity of the actual language pair (Kudugunta et al., 2019). In this work, we contribute to this research endeavor by proposing a suite of spectral-based measures that capture the degree of isomorphism (Søgaard et al., 2018) between the monolingual embedding spaces of two languages. Our main hypothesis is that the potential to align two em- bedding spaces and learn transfer functions can be estimated through the differences between the monolingual embeddings’ spectra. We therefore discuss representative statistics of the spectrum of an embedding space (i.e., the set of the singular values of the embedding matrix), such as its condi- tion number or its sorted list of singular values. We …

Content truncated to 500 words