Scalable Cross-Lingual Transfer of Neural Sentence Embeddings

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 51–60 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 51 Scalable Cross-Lingual Transfer of Neural Sentence Embeddings Hanan Aldarmaki1 and Mona Diab1,2 1The George Washington University 2AWS, Amazon AI aldarmaki@gwu.edu,diabmona@amazon.com Abstract We develop and investigate several cross- lingual alignment approaches for neural sen- tence embedding models, such as the super- vised inference classifier, InferSent, and se- quential encoder-decoder models. We eval- uate three alignment frameworks applied to these models: joint modeling, representation transfer learning, and sentence mapping, using parallel text to guide the alignment. Our re- sults support representation transfer as a scal- able approach for modular cross-lingual align- ment of neural sentence embeddings, where we observe better performance compared to joint models in intrinsic and extrinsic evalua- tions, particularly with smaller sets of parallel data. 1 Introduction Probabilistic sentence representation models gen- erally fall into two categories: bottom-up com- positional models, where sentence embeddings are composed from word embeddings via a lin- ear function like averaging, and top-down compo- sitional models that are trained with a sentence- level objective, typically within a neural architec- ture. Sequential data like sentences can be mod- eled using recurrent, recursive, or convolutional networks, which can implicitly learn intermediate sentence representations suitable for each learn- ing task. Depending on the training objective, these intermediate representations sometimes en- code enough semantic and syntactic features to be suitable as general-purpose sentence embed- dings. For examples, it was shown in Conneau et al. (2017a) that a model trained to maximize in- ference classification accuracy can yield generic representations that perform well across a wide set of extrinsic classification benchmarks. Other training objectives, like denoising auto-encoders or neural sequence to sequence models (Hill et al., 2016), can also yield general-purpose representa- tions with different characteristics. While bottom- up models can achieve superior performance in tasks that are independent of syntax, such as topic categorization, neural models often yield repre- sentations that encode syntactic and positional fea- tures, which results in superior performance in tasks that rely on sentence structure (Aldarmaki and Diab, 2018). General-purpose sentence embeddings can be used as features in various classification tasks, or to directly assess the similarity of a pair of sen- tences using the cosine measure. It is often de- sired to generalize word and sentence embeddings across several languages to facilitate cross-lingual transfer learning (Zhou et al., 2016) and mining of parallel sentences (Guo et al., 2018). For word em- beddings, cross-lingual learning can be achieved in various ways (Upadhyay et al., 2016), such as learning directly with a cross-lingual objective (Shi et al., 2015) or post-hoc alignment of mono- lingual word embeddings using dictionaries (Am- mar et al., 2016), parallel corpora (Gouws et al., 2015; Klementiev et al., 2012), or even with no bilingual supervision (Conneau et al., 2017b; Al- darmaki et al., 2018). For bottom-up composition like vector averaging, word-level alignment is suf- ficient to yield cross-lingual sentence embeddings. For top-down sentence embeddings, the efforts in …

Content truncated to 500 words