Revisiting the Context Window for Cross-lingual Word Embeddings

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 995–1005 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 995 Revisiting the Context Window for Cross-lingual Word Embeddings Ryokan Ri and Yoshimasa Tsuruoka The University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan {li0123,tsuruoka}@logos.t.u-tokyo.ac.jp Abstract Existing approaches to mapping-based cross- lingual word embeddings are based on the as- sumption that the source and target embedding spaces are structurally similar. The structures of embedding spaces largely depend on the co- occurrence statistics of each word, which the choice of context window determines. Despite this obvious connection between the context window and mapping-based cross-lingual em- beddings, their relationship has been underex- plored in prior work. In this work, we provide a thorough evaluation, in various languages, domains, and tasks, of bilingual embeddings trained with different context windows. The highlight of our findings is that increasing the size of both the source and target window sizes improves the performance of bilingual lexicon induction, especially the performance on fre- quent nouns. 1 Introduction Cross-lingual word embeddings can capture word semantics invariant among multiple languages, and facilitate cross-lingual transfer for low- resource languages (Ruder et al., 2019). Recent research has focused on mapping-based methods, which find a linear transformation from the source to target embedding spaces (Mikolov et al., 2013b; Artetxe et al., 2016; Lample et al., 2018). Learn- ing a linear transformation is based on a strong as- sumption that the two embedding spaces are struc- turally similar or isometric. The structure of word embeddings heavily de- pends on the co-occurrence information of words (Turney and Pantel, 2010; Baroni et al., 2014), i.e., word embeddings are computed by counting other words that appear in a specific context window of each word. The choice of context window changes the co-occurrence statistics of words and thus is crucial to determine the structure of an embedding space. For example, it has been known that an embedding space trained with a smaller linear window captures functional similar- ities, while a larger window captures topical simi- larities (Levy and Goldberg, 2014a). Despite this important relationship between the choice of con- text window and the structure of embedding space, how the choice of context window affects the struc- tural similarity of two embedding spaces has not been fully explored yet. In this paper, we attempt to deepen the under- standing of cross-lingual word embeddings from the perspective of the choice of the context win- dow through carefully designed experiments. We experiment with a variety of settings, with differ- ent domains and languages. We train monolingual word embeddings varying the context window sizes, align them with a mapping-based method, and then evaluate them with both intrinsic and downstream cross-lingual transfer tasks. Our re- search questions and the summary of the findings are as follows: RQ1: What kind of context windows produces a better alignment of two embedding spaces? Our result shows that increasing the window sizes of both the source and target embeddings improves the accuracy of …

Content truncated to 500 words