Context-Aware Cross-Lingual Mapping

Proceedings of NAACL-HLT 2019, pages 3906–3911 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3906 Context-Aware Cross-Lingual Mapping Hanan Aldarmaki1 and Mona Diab1,2 1The George Washington University 2AWS, Amazon AI aldarmaki@gwu.edu,diabmona@amazon.com Abstract Cross-lingual word vectors are typically ob- tained by fitting an orthogonal matrix that maps the entries of a bilingual dictionary from a source to a target vector space. Word vectors, however, are most commonly used for sen- tence or document-level representations that are calculated as the weighted average of word embeddings. In this paper, we propose an alternative to word-level mapping that bet- ter reflects sentence-level cross-lingual simi- larity. We incorporate context in the transfor- mation matrix by directly mapping the aver- aged embeddings of aligned sentences in a par- allel corpus. We also implement cross-lingual mapping of deep contextualized word embed- dings using parallel sentences with word align- ments. In our experiments, both approaches resulted in cross-lingual sentence embeddings that outperformed context-independent word mapping in sentence translation retrieval. Fur- thermore, the sentence-level transformation could be used for word-level mapping without loss in word translation quality. 1 Introduction Cross-lingual word vector models aim to embed words from multiple languages into a shared vec- tor space to enable cross-lingual transfer and dic- tionary expansion (Upadhyay et al., 2016). One of the most common and effective approaches for obtaining bilingual word embeddings is by fitting a linear transformation matrix on the entries of a bilingual seed dictionary (Mikolov et al., 2013). This approach is versatile and scalable: multilin- gual embeddings can be obtained by mapping the vector spaces of multiple languages into a shared target language, typically English. In addition, imposing an orthogonality constraint on the map- ping ensures that the original pair-wise distances are preserved after the transformation and results in better word translation retrieval (Artetxe et al., 2016; Smith et al., 2017). While word vector spaces tend to be globally consistent across language variations (Aldarmaki et al., 2018), individual words like homographs with unrelated senses (e.g. ‘bank’, ‘coast’) and phrasal verbs (‘stand up’, ‘stand out’) are likely to behave less consistently in multilingual vector spaces due to their different usage distributions. Consequently, using such words in the alignment dictionary may result in suboptimal overall map- ping. We propose two approaches to counteract this effect by incorporating sentential context in the mapping process without explicit word sense disambiguation or additional linguistic resources. The first approach is based on the recently pro- posed contextualized embeddings from language models, ELMo (Peters et al., 2018). Using a parallel corpus with word-alignments, we extract contextualized embeddings to construct a context- aware dictionary for mapping. The second ap- proach is to learn a transformation between sen- tence embeddings rather than individual word em- beddings. Since these embeddings include context that spans full sentences, we surmise that a map- ping learned at this level would be more robust to individual word misalignments. We used a constrained set of parallel sentences ranging from one hundred to a million sentences for alignment. …

Content truncated to 500 words