Pre-tokenization of Multi-word Expressions in Cross-lingual Word Embeddings

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4451–4464, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4451 Pre-tokenization of Multi-word Expressions in Cross-lingual Word Embeddings Naoki Otani1 Satoru Ozaki1 Xingyuan Zhao1 Yucen Li2∗ Micaelah St Johns3∗ Lori Levin1 1Carnegie Mellon University 2Facebook 3Stanford University 1{notani,sozaki,xingyuaz,levin}@andrew.cmu.edu 2yucenli@fb.com 3mstjohns@stanford.edu Abstract Cross-lingual word embedding (CWE) algo- rithms represent words in multiple languages in a unified vector space. Multi-Word Expres- sions (MWE) are common in every language. When training word embeddings, each com- ponent word of an MWE gets its own sepa- rate embedding, and thus, MWEs are not trans- lated by CWEs. We propose a simple method for word translation of MWEs to and from En- glish in ten languages: we first compile lists of MWEs in each language and then tokenize the MWEs as single tokens before training word embeddings. CWEs are trained on a word- translation task using the dictionaries that only contain single words. In order to evaluate MWE translation, we created bilingual word lists from multilingual WordNet that include single-token words and MWEs, and most im- portantly, include MWEs that correspond to single words in another language. We show that the pre-tokenization of MWEs as single tokens performs better than averaging the em- beddings of the individual tokens of the MWE. We can translate MWEs at a top-10 precision of 30-60%. The tokenization of MWEs makes the occurrences of single words in a train- ing corpus more sparse, but we show that it does not pose negative impacts on single-word translations. 1 Introduction Cross-lingual word embeddings (CWEs) are real- valued vector representations of words in multiple languages placed in a shared vector space, with the intention that words with closer meanings have closer locations in the vector space. First, mono- lingual word embeddings are trained based on the hypothesis of distributional semantics (Harris, 1954) that context approximates meaning. They ∗This work was conducted while YL and MJ were at Carnegie Mellon University. are learned from data in a way that words used in similar contexts have similar vectors. Following that, the monolingual word embeddings are aligned to produce CWEs. CWEs are an essential build- ing block in modern cross-lingual methods and can also be used to induce bilingual lexicons from a small seed dictionary (Mikolov et al., 2013). An important and overlooked fact is that before CWEs are trained, the corpus is pre-processed by a word tokenizer. This illustrates a clear limitation of the state-of-the-art CWEs: they can only align words that happen to be considered as single tokens by the word tokenizer. Multi-word expressions (MWEs) are combina- tions of orthographic words, whose meaning, form, use, or distribution is non-compositional or un- predictable in some way (Sag et al., 2002; Bald- win and Kim, 2010). They come in diverse forms such as compound nouns (dance floor), named en- tities (United States), phrasal verbs (give up), and connectives (as well as). Word tokenizers do not recognize MWEs as single units but rather as a …

Content truncated to 500 words