Polyglot: Distributed Word Representations for Multilingual NLP

Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192, Sofia, Bulgaria, August 8-9 2013. c⃝2013 Association for Computational Linguistics Polyglot: Distributed Word Representations for Multilingual NLP Rami Al-Rfou’ Bryan Perozzi Computer Science Dept. Stony Brook University Stony Brook, NY 11794 {ralrfou, bperozzi, skiena}@cs.stonybrook.edu Steven Skiena Abstract Distributed word representations (word embeddings) have recently contributed to competitive performance in language modeling and several NLP tasks. In this work, we train word embeddings for more than 100 languages using their cor- responding Wikipedias. We quantitatively demonstrate the utility of our word em- beddings by using them as the sole fea- tures for training a part of speech tagger for a subset of these languages. We find their performance to be competitive with near state-of-art methods in English, Dan- ish and Swedish. Moreover, we inves- tigate the semantic features captured by these embeddings through the proximity of word groupings. We will release these embeddings publicly to help researchers in the development and enhancement of mul- tilingual applications. 1 Introduction Building multilingual processing systems is a challenging task. Every NLP task involves dif- ferent stages of preprocessing and calculating in- termediate representations that will serve as fea- tures for later stages. These stages vary in com- plexity and requirements for each individual lan- guage. Despite recent momentum towards devel- oping multilingual tools (Nivre et al., 2007; Hajiˇc et al., 2009; Pradhan et al., 2012), most of NLP research still focuses on rich resource languages. Common NLP systems and tools rely heavily on English specific features and they are infrequently tested on multiple datasets. This makes them hard to port to new languages and tasks (Blitzer et al., 2006). A serious bottleneck in the current approach for developing multilingual systems is the require- ment of familiarity with each language under con- sideration. These systems are typically carefully tuned with hand-manufactured features designed by experts in a particular language. This approach can yield good performance, but tends to create complicated systems which have limited portabil- ity to new languages, in addition to being hard to enhance and maintain. Recent advancements in unsupervised feature learning present an intriguing alternative. In- stead of relying on expert knowledge, these ap- proaches employ automatically generated task- independent features (or word embeddings) given large amounts of plain text. Recent developments have led to state-of-art performance in several NLP tasks such as language modeling (Bengio et al., 2006; Mikolov et al., 2010), and syntactic tasks such as sequence tagging (Collobert et al., 2011). These embeddings are generated as a result of training “deep” architectures, and it has been shown that such representations are well suited for domain adaptation tasks (Glorot et al., 2011; Chen et al., 2012). We believe two problems have held back the research community’s adoption of these methods. The first is that learning representations of words involves huge computational costs. The process usually involves processing billions of words over weeks. The second is that so far, these systems have been built and tested mainly on English. …

Content truncated to 500 words