Investigating Cross-Lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 33–43 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 33 Investigating Cross-lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation Qianchu Liu, Diana McCarthy, Ivan Vuli´c, Anna Korhonen Language Technology Lab, University of Cambridge English Faculty Building, 9 West Road, Cambridge CB3 9DA, United Kingdom {ql261,iv250,alk23}@cam.ac.uk, diana@dianamccarthy.co.uk Abstract In this paper, we present a thorough inves- tigation on methods that align pre-trained contextualized embeddings into shared cross- lingual context-aware embedding space, pro- viding strong reference benchmarks for fu- ture context-aware crosslingual models. We propose a novel and challenging task, Bilin- gual Token-level Sense Retrieval (BTSR). It specifically evaluates the accurate alignment of words with the same meaning in cross- lingual non-parallel contexts, currently not evaluated by existing tasks such as Bilingual Contextual Word Similarity and Sentence Re- trieval. We show how the proposed BTSR task highlights the merits of different align- ment methods. In particular, we find that us- ing context average type-level alignment is ef- fective in transferring monolingual contextual- ized embeddings cross-lingually especially in non-parallel contexts, and at the same time im- proves the monolingual space. Furthermore, aligning independently trained models yields better performance than aligning multilingual embeddings with shared vocabulary. 1 Introduction Contextualized embeddings have been shown to achieve superior performance compared to static word embeddings in English (Peters et al., 2018; Devlin et al., 2019). Despite recent efforts to better understand their multilingual variants (Pires et al., 2019), leveraging these available pretrained contex- tualized embeddings to learn cross-lingual contex- tualized embeddings is still an under-explored area: past cross-lingual embedding alignment methods have mainly focused on static embeddings (Ruder et al., 2019). In this paper, we introduce a first study that investigates and compares different ways of aligning the pretrained contextualized embeddings. In particular, we make the comparisons focused on the following properties: (1) aligning contextual- ized embeddings at the level of word tokens versus word types; (2) different training signals: static dic- tionaries, word alignment, or sentence alignment from parallel data; and (3) aligning different model variants: aligning from independently trained mod- els versus aligning embeddings from a multilingual model with shared vocabulary. We evaluate the methods on a variety of context- aware tasks. Besides two previously established evaluation tasks (1) Bilingual Contextual Word Similarity (Chi and Chen, 2018) and (2) Sentence Retrieval (Conneau et al., 2017), we introduce a new task: Bilingual Token-level Sense Retrieval (BTSR). It is more challenging than the alterna- tives as it requires the accurate cross-lingual re- trieval of contextualized words on the token level which are disambiguated both in the source and the target language using non-parallel contexts. We provide BTSR task data and run evaluations on two language pairs: English–Chinese (EN–ZH) and English–Spanish (EN–ES). The data and guide- lines can be found at: https://github.com/ qianchu/BTSR Our main findings are as follows. (1) Using the average of the contextualized word represen- tations as type-level anchors is effective and ro- bust for aligning pre-trained contextualized em- beddings …

Content truncated to 500 words