Cross-Lingual Word Embeddings for Morphologically Rich Languages

Proceedings of Recent Advances in Natural Language Processing, pages 1222–1228, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_140 1222 Cross-Lingual Word Embeddings for Morphologically Rich Languages Ahmet ¨Ust¨un Gosse Bouma Gertjan van Noord University of Groningen {a.ustun, g.bouma, g.j.m.van.noord}@rug.nl Abstract Cross-lingual word embedding models learn a shared vector space for two or more lan- guages so that words with similar meaning are represented by similar vectors regardless of their language. Although the existing mod- els achieve high performance on pairs of mor- phologically simple languages, they perform very poorly on morphologically rich languages such as Turkish and Finnish. In this pa- per, we propose a morpheme-based model in order to increase the performance of cross- lingual word embeddings on morphologically rich languages. Our model includes a sim- ple extension which enables us to exploit mor- phemes for cross-lingual mapping. We ap- plied our model for the Turkish-Finnish lan- guage pair on the bilingual word translation task. Results show that our model outper- forms the baseline models by 2% in the nearest neighbour ranking. 1 Introduction Cross-lingual word embeddings (CLEs) have drawn a lot of attention in recent times. CLE models learn vectors of words in two or more languages and represent them in a shared cross- lingual word embedding space, where words with similar meaning have similar vectors, independent of their language. Most popular approaches for CLEs are mapping-based approaches which are also called offline approaches. These kinds of approaches require only pre-trained monolingual embeddings and a small seed dictionary so that the CLE model learns a mapping that minimizes the distance between word pairs in the seed dictionary to align the pre-trained embedding spaces. CLE models enable multi-lingual modeling which has direct applications on cross-lingual tasks such as unsupervised machine translation (Lample et al., 2017), and cross-lingual trans- fer for downstream NLP tasks and low-resource languages. Document classification (Klementiev et al., 2012), information retrieval (Vuli´c and Moens, 2015), dependency parsing (Guo et al., 2015), and sequence labelling (Zhang et al., 2016) are examples of downstream NLP tasks in which CLEs serve as a source of cross-lingual knowl- edge. Although the existing models achieve high per- formance, agglutinative languages, such as Turk- ish, Finnish and Estonian, pose a challenge to learn cross-lingual word embeddings due to three main reasons. First, with respect to the mono- lingual aspects, morphological complexity causes high sparsity which decreases the quality of mono- lingual embedding spaces (Cao and Rei, 2016; ¨Ust¨un et al., 2018). Second, in the context of CLEs, the rich morphology causes inaccu- rate mappings especially for complex words be- cause the existing CLE models cannot access the sub-word level information to align complex words with the correct morphological counter- parts. Søgaard et al. (2018) shows that the ex- isting CLE models underperform on rich mor- phological complexity. On the bilingual dictio- nary induction task, while the baseline method achieves 82.62% score on English-Spanish, it per- forms very poorly on English-Finnish (28.01%), English-Estonian (31.45%) and English-Turkish (39.22%). In the Estonian-Finnish dictionary in- duction experiment in …

Content truncated to 500 words