Are Language-Agnostic Sentence Representations Actually Language-Agnostic?

Proceedings of Recent Advances in Natural Language Processing, pages 274–280 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_032 274 Are Language-Agnostic Sentence Representations actually Language-Agnostic? Yu Chen Tania Augustinova Department of Language Science and Technology Saarland University, Germany {yuchen,tania}@coli.uni-saarland.de Abstract With the emergence of pre-trained multilin- gual models, multilingual embeddings have been widely applied in various natural lan- guage processing tasks. Language-agnostic models provide a versatile way to convert lin- guistic units from different languages into a shared vector representation space. The rel- evant work on multilingual sentence embed- dings has reportedly reached low error rate in cross-lingual similarity search tasks. In this pa- per, we apply the pre-trained embedding mod- els and the cross-lingual similarity search task in diverse scenarios, and observed large dis- crepancy in results in comparison to the orig- inal paper. Our findings on cross-lingual sim- ilarity search with different newly constructed multilingual datasets show not only correlation with observable language similarities but also strong influence from factors such as transla- tion paths, which limits the interpretation of the language-agnostic property of the LASER model. 1 Introduction Multilingual joint embeddings map language units from different languages into the same embedding space in order to make them comparable, which fa- cilitates cross-lingual transfer. Being essential for building NLP models of low resourced languages, such an integrated representation is also useful for cross-lingual tasks like machine translation, espe- cially when multiple languages are involved or there is a lack of appropriate data. While word embeddings are widely used in NLP tasks, sentence representations become quite impor- tant for capturing underlying semantic relations in texts across different languages. Hence, instead of simply pooling word representation together, vari- ous neural network methods have proposed to pro- duce more coherent sentence representations. Re- cent advances in multilingual sentence embedding modeling (Schwenk and Douze, 2017; Feng et al., 2020; Hirota et al., 2020) begin to show strong per- formance on many multilingual NLP tasks, but it does not always work equally well for all languages. We repeat the cross-lingual similarity search task with LASER (Schwenk and Douze, 2017) with more challenging corpora in order to identify what affects the actual performance. Based on our find- ings, we propose directions for future development of such models. 2 LASER LASER (Language-Agnostic SEntence Represen- tations) (Schwenk and Douze, 2017; Artetxe and Schwenk, 2019) is contextualized language model based on a BiLSTM encoder trained using a translation objective on parallel data from Eu- roparl (Koehn, 2005), United Nations (Ziemski et al., 2016), OpenSubtitles2018 (Lison et al., 2018), Global Voices (Prokopidis et al., 2016), Tanzil and Tatoeba mostly available on the OPUS website (Tiedemann, 2012). The LASER model is able to handle 93 different languages. Also Schwenk and Douze (2017) has proposed a similarity-search-based framework in order to eval- uate multilingual joint representations. With a col- lection of S parallel sentences for a given language pair, a multilingual similarity search is performed for the closest target sentence for each of the source sentences, and an error is counted if it is …

Content truncated to 500 words