KIT-Multi: A Translation-Oriented Multilingual Embedding Corpus

KIT-Multi: A Translation-Oriented Multilingual Embedding Corpus Thanh-Le Ha, Jan Niehues, Matthias Sperber, Ngoc Quan Pham and Alexander Waibel Karlsruhe Institute of Technology, Karlsruhe, Germany firstname.lastname@kit.edu Abstract Cross-lingual word embeddings are the representations of words across languages in a shared continuous vector space. Cross-lingual word embeddings have been shown to be helpful in the development of cross-lingual natural language processing tools. In case of more than two languages involved, we call them multilingual word embeddings. In this work, we introduce a multilingual word embedding corpus which is acquired by using neural machine translation. Unlike other cross-lingual embedding corpora, the embeddings can be learned from significantly smaller portions of data and for multiple languages at once. An intrinsic evaluation on monolingual tasks shows that our method is fairly competitive to the prevalent methods but on the cross-lingual document classification task, it obtains the best figures. We are in the process to produce the embeddings for more languages, especially the languages which belong to the same family or sematically close to each others, such as Japanese-Korean, Chinese-Vietnamese, German-Dutch, or Latin-based languagues. Furthermore, the corpus is being analyzedd regarding its usage and usefulness in other cross-lingual tasks. Keywords: multilingual embeddings, cross-lingual embeddings, neural machine translation, multi-source translation 1. Introduction Inducing cross-lingual word embeddings is essentially acquiring word embeddings in different languages. The cross-lingual word embeddings can then be used as pre- trained models in cross-lingual applications such as cross- lingual document classification, information retrieval, tex- tual entailment and question answering. Cross-lingual word embeddings can also help to perform transfer learn- ing from a well-resource language to another low-resource language on various tasks, e.g. in building WordNet or an- notating semantic relations. There have been various methods of cross-lingual em- bedding induction being proposed, but most of them are essentially bilingual in the perspective that they learn to in- duce bilingual embeddings from bilingual data 1. Basically these methods optimize some cross-lingual constraints so that the semantic similarity between words corresponds to the closeness of these representations in a common vec- tor space. Consequently, if they need cross-lingual em- beddings for a new language pair, they must apply their inducing method on that new bilingual data. Furthermore, there would be some domain mismatch between the new ac- quired embeddings and the others if the new bilingual data are from different domain. The aforementioned limitations of those cross-lingual corpora motivates us to design a mul- tilingual embedding inducing method from a single corpus which is available in as many languages as possible. In this paper, we propose such an approach utiliz- ing a multilingual neural machine translation (NMT) sys- tem to constrain the embeddings from n source languages while translating into the same target language (as we call it multi-source NMT). The source embeddings employed in this model are implicitly forced to learn the common semantic regularities in order to maximize the translation quality of every language pair in the system. Once the 1For a thoroughly review of the most popular and advan- tageous techniques of …

Content truncated to 500 words