Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings

Proceedings of the 29th International Conference on Computational Linguistics, pages 4381–4396 October 12–17, 2022. 4381 Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings Linlin Liu12∗ Thien Hai Nguyen1 Shafiq Joty2 Lidong Bing1† Luo Si1 1DAMO Academy, Alibaba Group 2Nanyang Technological University, Singapore {linlin.liu, thienhai.nguyen, l.bing, luo.si}@alibaba-inc.com srjoty@ntu.edu.sg Abstract Cross-lingual word embeddings (CLWE) have been proven useful in many cross-lingual tasks. However, most existing approaches to learn CLWE including the ones with contextual em- beddings are sense agnostic. In this work, we propose a novel framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionar- ies only. We operationalize our framework by first proposing a novel sense-aware cross entropy loss to model word senses explicitly. The monolingual ELMo and BERT models pre- trained with our sense-aware cross entropy loss demonstrate significant performance improve- ment for word sense disambiguation tasks. We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross- lingual model pretraining, and pretrain cross- lingual models for several language pairs (En- glish to German/Spanish/Japanese/Chinese). Compared with the best baseline results, our cross-lingual models achieve 0.52%, 2.09% and 1.29% average performance improvements on zero-shot cross-lingual NER, sentiment clas- sification and XNLI tasks, respectively. 1 1 Introduction Cross-lingual word embeddings (CLWE) provide a shared representation space for knowledge transfer between languages, yielding state-of-the-art perfor- mance in many cross-lingual natural language pro- cessing (NLP) tasks. Most of the previous works have focused on aligning static embeddings. To utilize the richer information captured by the pre- trained language model, more recent approaches attempt to extend previous methods to align con- textual representations. ∗ Linlin Liu is under the Joint PhD Program between Alibaba and Nanyang Technological University. †Corresponding author. 1Our code is available at https://github.com/ ntunlp/multisense_embedding_alignment. git. Aligning the dynamic and complex contextual spaces poses significant challenges, so most of the existing approaches only perform coarse-grained alignment. Schuster et al. (2019) compute the aver- age of contextual embeddings for each word as an anchor, and then learn to align the static anchors using a bilingual dictionary. In another work, Al- darmaki and Diab (2019) use parallel sentences in their approach, where they compute sentence representations by taking the average of contextual word embeddings, and then they learn a projection matrix to align sentence representations. They find that the learned projection matrix also works well for word-level NLP tasks. Besides, unsupervised multilingual language models (Devlin et al., 2018; Artetxe and Schwenk, 2019; Conneau et al., 2019; Liu et al., 2020) pretrained on multilingual cor- pora have also demonstrated strong cross-lingual transfer performance. However, studies (Wang et al., 2020; Cao et al., 2020; Efimov et al., 2022; Tien and Steinert-Threlkeld, 2022) have shown that adjusting the unsupervised multilingual language model with parallel sentences can help further im- prove cross-lingual performance. Though contextual word embeddings are in- tended to provide different representations of the same word in distinct contexts, Schuster et al. (2019) find that the contextual embeddings of dif- ferent senses of one word …

Content truncated to 500 words