Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Proceedings of NAACL-HLT 2019, pages 1599–1613 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1599 Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing Tal Schuster∗,1, Ori Ram∗,2, Regina Barzilay1, Amir Globerson2 1Computer Science and Artificial Intelligence Lab, MIT 2Tel Aviv University {tals, regina}@csail.mit.edu, {ori.ram, gamir}@cs.tau.ac.il Abstract We introduce a novel method for multilin- gual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counter- parts, aligning them poses a challenge due to their dynamic nature. To this end, we con- struct context-independent variants of the orig- inal monolingual spaces and utilize their map- ping to derive an alignment for the context- dependent spaces. This mapping readily sup- ports processing of a target language, improv- ing transfer by context-aware embeddings. Our experimental results demonstrate the ef- fectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outper- forms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average.1 1 Introduction Multilingual embedding spaces have been demon- strated to be a promising means for enabling cross- lingual transfer in many natural language pro- cessing tasks (e.g. Ammar et al. (2016); Lample et al. (2018)). Similar to how universal part-of- speech tags enabled parsing transfer across lan- guages (Petrov et al., 2012), multilingual word embeddings further improve transfer capacity by enriching models with lexical information. Since this lexical representation is learned in an un- supervised fashion and thus can leverage large amounts of raw data, it can capture a more nu- anced representation of meaning than unlexical- ized transfer. Naturally, this enrichment is trans- ∗ Equal contribution 1Code and models: https://github.com/ TalSchuster/CrossLingualELMo. lated into improved transfer accuracy, especially in low-resource scenarios (Guo et al., 2015). In this paper, we are moving further along this line and exploring the use of contextual word embeddings for multilingual transfer. By dy- namically linking words to their various con- texts, these embeddings provide a richer se- mantic and syntactic representation than tradi- tional context-independent word embeddings (Pe- ters et al., 2018). A straightforward way to utilize this richer representation is to directly apply exist- ing transfer algorithms on the contextual embed- dings instead of their static counterparts. In this case, however, each token pair is represented by many different vectors corresponding to its spe- cific context. Even when supervision is available in the form of a dictionary, it is still unclear how to utilize this information for multiple contextual embeddings that correspond to a word translation pair. In this paper, we propose a simple but effec- tive mechanism for constructing a multilingual space of contextual embeddings. Instead of learn- ing the alignment in the original, complex con- textual space, we drive the mapping process us- ing context-independent embedding anchors. We obtain these anchors by factorizing the contextual embedding space into context-independent and context-dependent parts. Operating at the anchor …

Content truncated to 500 words