Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6456 - 6472 December 7-11, 2022 ©2022 Association for Computational Linguistics Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation∗ Deng Cai♥ Xin Li♠,† Jackie Chun-Sing Ho♥ Lidong Bing♠ Wai Lam♥ ♥The Chinese University of Hong Kong ♠DAMO Academy, Alibaba Group thisisjcykcd@gmail.com {xinting.lx,l.bing}@alibaba-inc.com {jackieho@link,wlam@se}.cuhk.edu.hk Abstract We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR). Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. It also helps reduce surface variations across different expressions and languages. Unlike most prior work that only evaluates the abil- ity to measure semantic similarity, we present a thorough evaluation of existing multilin- gual sentence embeddings and our improved versions, which include a collection of five transfer tasks in different downstream applica- tions. Experiment results show that retrofitting multilingual sentence embeddings with AMR leads to better state-of-the-art performance on both semantic textual similarity and transfer tasks. Our codebase and evaluation scripts can be found at https://github.com/jcyk/ MSE-AMR. 1 Introduction Multilingual sentence embedding (MSE) aims to provide universal sentence representations shared across different languages (Hermann and Blun- som, 2014; Pham et al., 2015; Schwenk and Douze, 2017). As an important ingredient of cross- lingual and multilingual natural language process- ing (NLP), MSE has recently attracted increasing attention in the NLP community. MSE has been widely adopted to bridge the language barrier in several downstream applications such as bitext min- ing (Guo et al., 2018; Schwenk, 2018), document classification (Eriguchi et al., 2018; Singla et al., 2018; Yu et al., 2018) and natural language infer- ence (Artetxe and Schwenk, 2019). Prior work typ- ically borrows fixed-size embedding vectors from ∗ This work was supported by Alibaba Group through the Alibaba Innovative Research (AIR) Program. † XL is the corresponding author. multilingual neural machine models (Schwenk and Douze, 2017; Yu et al., 2018) or trains siamese neural networks to align the semantically similar sentences written in different languages (Wieting et al., 2019; Yang et al., 2020; Feng et al., 2020). Despite the recent progress, the current evalua- tion of multilingual sentence embeddings has fo- cused on cross-lingual Semantic Textual Similar- ity (STS) (Agirre et al., 2016; Cer et al., 2017) or bi-text mining tasks (Zweigenbaum et al., 2018; Artetxe and Schwenk, 2019). Nevertheless, as pointed out by Gao et al. (2021), the evaluation on semantic similarity may not be sufficient be- cause better performance on STS does not always indicate better embeddings for downstream tasks. Therefore, for a more comprehensive MSE evalua- tion, it is necessary to additionally evaluate down- stream tasks, which is largely ignored in recent work (Chidambaram et al., 2019; Reimers and Gurevych, 2020; Feng et al., 2020). In this pa- per, we collect a set of multilingual transfer tasks and test various existing multilingual sentence em- beddings. We find that different methods excel at different tasks and the conclusions drawn from …

Content truncated to 500 words