Wiktionary-based word embeddings

Wiktionary-Based Word Embeddings Gerard de Melo gdm@demelo.org IIIS, Tsinghua University, Beijing, China Abstract Vectorial representations of words have grown remarkably popular in natural language process- ing and machine translation. The recent surge in deep learning-inspired methods for producing distributed representations has been widely noted even outside these fields. Existing repre- sentations are typically trained on large monolingual corpora using context-based prediction models. In this paper, we propose extending pre-existing word representations by exploiting Wiktionary. This process results in a substantial extension of the original word vector repre- sentations, yielding a large multilingual dictionary of word embeddings. We believe that this resource can enable numerous monolingual and cross-lingual applications, as evidenced in a series of monolingual and cross-lingual semantic experiments that we have conducted. 1 Introduction Vectorial representations of words have grown to play an important role in natural language processing and machine translation. Especially for the latter, deep learning and representation learning-based approaches have recently proven remarkably effective (Sutskever et al., 2014; Luong et al., 2015). Vector-based encodings of meaning are a central ingredient in many of these recent neural machine translation systems, although they can also be beneficial in ordinary phrase-based machine translation (Mikolov et al., 2013b). In this work, we focus on the task of creating vector representations of multilingual words (as well as lexicalized phrases). Previous work in this area has relied on multilingual corpora to train bilingual word vectors. We investigate to what extent external large-scale resources can be used to create much more multilingual word representation data. In particular, we rely on Wiktionary, a sister project of Wikipedia that for many years now has been creating a large, col- laboratively edited online dictionary. Due to its rich multilingual data, now with over 4 million entries in over 1,000 languages, Wiktionary has been used extensively in natural language pro- cessing, e.g. for part-of-speech tagging (Li et al., 2012) and named entity recognition (Richman and Schone, 2008), for cross-language image search (Etzioni et al., 2007) and text classification (Nastase and Strapparava, 2013), and for producing language registries (de Melo, 2015) and etymological databases (de Melo, 2014). Wiktionary has also made it possible to translate lex- ical knowledge bases such as WordNet to hundreds of languages (de Melo and Weikum, 2009) and to translate thesauri (Borin et al., 2014). Finally, it has been used as an extra ingredient in regular machine translation systems (G¨ohring, 2014). Relying on Wiktionary instead of on other training data has playfully been called Wiki- ly supervision (Li et al., 2012). Our work constitutes a form of Wiktionary-based supervision This research was supported by China 973 Program Grants 2011CBA00300, 2011CBA00301, and NSFC Grants 61033001, 61361136003, 61450110088. Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 346 for multilingual word representation learning. More specifically, our method starts with ex- isting word representations such as the widely available ones trained on large English corpora (Mikolov et al., 2013a; Pennington et al., 2014). It then uses Wiktionary to …

Content truncated to 500 words