„Mann“ is to “Donna” as「国王」is to « Reine » Adapting the Analogy Task for Multilingual and Contextual Embeddings

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 270–283 July 13-14, 2023 ©2023 Association for Computational Linguistics „Mann“ is to “Donna” as「国王」is to « Reine » Adapting the Analogy Task for Multilingual and Contextual Embeddings Timothee Mickus♠ Eduardo Calò♡ Léo Jacqmin♢ Denis Paperno♡ Mathieu Constant♣ ♠University of Helsinki, timothee.mickus@helsinki.fi ♡Utrecht University, {e.calo,d.paperno}@uu.nl ♢Orange Labs, leo.jacqmin@orange.com ♣ATILF, CNRS/Université de Lorraine, mconstant@atilf.fr Abstract How does the word analogy task fit in the mod- ern NLP landscape? Given the rarity of com- parable multilingual benchmarks and the lack of a consensual evaluation protocol for contex- tual models, this remains an open question. In this paper, we introduce MATS: a multilingual analogy dataset, covering forty analogical rela- tions in six languages, and evaluate human as well as static and contextual embedding perfor- mances on the task. We find that not all ana- logical relations are equally straightforward for humans, static models remain competitive with contextual embeddings, and optimal settings vary across languages and analogical relations. Several key challenges remain, including creat- ing benchmarks that align with human reason- ing and understanding what drives differences across methodologies. https://github.com/ATILF-UMR7118/MATS 1 Introduction Ever since the work of Mikolov et al. (2013b), anal- ogy solving has been a staple of public outreach in NLP: It has been featured both in science communi- cation1 and in the classroom.2 This task consists in finding a target word b2, given a cue word b1 it is related to, and another pair of words a1 and b2 that express the same relation. For example, we can ask what is the word that relates to “king” in the same manner that “woman” relates to “man”: This target ought to be “queen”. The introduction of pre-trained contextualized embeddings (Peters et al., 2018) opened up a new research area where to expand prior knowledge about static models. This includes the analogy task. Suggestions have been put forward as to how 1E.g., it is discussed by the Computerphile YouTube chan- nel, cf. https://youtu.be/gQddtTdmG_8?t=662. 2To take an example, see the Winter 2017 NLP lectures at Stanford, https://youtu.be/ASn7ExxLZws?t=3257. to best adapt it: Ushio et al. (2021) propose to use a prompt-based method, whereas Vuli´c et al. (2020) and Lenci et al. (2022) try to derive static embeddings from BERT to fall back on the algo- rithm of Mikolov et al. (2013b). However, much work remains to be done to properly contrast and compare the performance of contextual and static embedding models on the analogy task. Another observation to be made is that reliable compar- isons across languages are rare. On the one hand, datasets for English—such as the GATS (Google Analogy Test Set, Mikolov et al., 2013a) and BATS (Balanced Analogy Test Set, Gladkova et al., 2016) benchmarks—have been adapted or translated to a wide variety of languages. On the other hand, approaches specifically focusing on establishing multilingual comparisons are, to our knowledge, limited to Grave et al. (2018), Ulˇcar et al. (2020), and Peng et al. (2022)—none of which considers …

Content truncated to 500 words